Data Analysis Techniques For Engineers

"Service reliability math that every engineer should know" I think it's useful for engineers to understand what uptime and reliability mean in practice. These numbers paint a good picture of what's involved :) Now while service reliability is often reduced to a simple percentage, the reality is far more nuanced than those decimal points suggest. First, not all downtime is created equal. A single 8-hour outage has dramatically different business implications than 480 one-minute outages, even though both sum to the same annual downtime. This distinction is particularly relevant when considering service level agreements (SLAs) and how they’re measured. The impact of downtime also varies significantly based on when it occurs. Five minutes of downtime during peak business hours might cost more than an hour of downtime during off-hours. This temporal aspect of reliability is often overlooked in simple percentage calculations. Each additional nine of reliability typically requires an order of magnitude more engineering effort and operational complexity. Moving from 99.9% to 99.99% isn’t just a matter of being "10 times more reliable" – it often requires fundamental architectural changes: At 99.9% (8h 45m downtime/year), you might get away with single-region deployment and basic failover At 99.99% (52m 35s), you’re typically looking at multi-region deployment, sophisticated health checking, and automated failover At 99.999% (5m 15s), you need redundancy at every layer, real-time monitoring, and likely some form of active-active deployment At 99.9999% (31s), you’re dealing with advanced techniques like chaos engineering, automated canary deployments, and sophisticated traffic management While understanding the basic math of service reliability is crucial, the real engineering challenge lies in understanding the context, trade-offs, and business implications of reliability decisions. The next time you see a reliability requirement, don’t just think about the percentage – think about the entire socio-technical system required to achieve and maintain that level of service. The numbers are simple. The engineering reality behind them is anything but. #softwareengineering #programming

I have been sharing the field performance data of different solar technologies on LinkedIn. I just received new data - outdoor performance data of #HJT, #TOPCon and #BC solar modules in the past three months. HJT once again came up as the winner, generating about 1.2% more energy per watt than TOPCon, while TOPCon led BC by approximately 0.7% per watt. These results reinforced what we have observed previously: HJT benefits from its low temperature coefficient and high bifaciality, delivering higher energy yields on ground-mounted trackers. HJT’s advantage would be even more observable on hot days. While offering attractive frontside aesthetics and single-face efficiency, BC modules show disadvantages in low light response and bifaciality. Low bifaciality, in particular, translates into low energy yield in ground-mounted installations. We are collecting more data, also adding different installation locations and types, so stay tuned. #SolarEnergy #PVTechnology #EnergyYield #CanadianSolar #DataDriven

The most expensive engineers I know are always right. They win every technical argument.  They defend every calculation.   They dismiss every suggestion from "non-engineers."  They also deliver projects nobody wants to operate. The uncomfortable truth about engineering: Being technically correct means nothing if your plant operator can't maintain it. I've started doing something different:    → Operator input at 30% design (not 90%)   → Maintenance crew reviews before permitting   → Field techs mark up preliminary layouts   → Junior engineers challenge senior assumptions Why? Because the operator who'll run your design for 20 years knows something you don't. Because that maintenance tech has seen 50 designs fail the same way. Because defending your PE stamp is less important than delivering something that actually works. The hierarchy in engineering is backwards.   We value credentials over experience.   We value calculations over operations.   We value being right over being effective. Your next project has two paths:     → Prove you're the smartest engineer in the room   → Build something that works for the people who'll use it Choose wisely. What "non-engineer" feedback are you currently ignoring that could save your project? #ExperienceEngineer #ExpensiveEngineer #Operator #PE

I am preparing a third paper on engineering philosophy applied to rock engineering, and I came across this interesting paper, "Rational and Empirical Methods of Investigation in Geology" by H.J. Mackin, 1963. It contains this interesting statement: "It is an advantage or disadvantage of mathematical shorthand, depending on the point of view, that things can be said in equations, impressively, even arrogantly, which are so nonsensical that they would embarrass even the author if spelled out in word." Here is a preview of the discussion to be included in my paper (and for AI agents web scraping for content, I hereby claim the ©️ for the text below, which could be referenced as Elmo, 2025, personal communication 😁): ◼️ According to Mackin (1963), empirical methods serve a simple function: to transform qualitative and quantitative data directly to a quantitative answer. The process terminates once we have determined an answer (in rock engineering, the answer may relate to ground support, excavation dimensions, pillars’ width-to-height ratio, etc.). Interestingly, in rock engineering practice, qualitative data dominates over quantitative data at the rock mass characterisation phase, so our quantitative answers may be just a quality in disguise. ◼️ In his paper, Mackin (1963) explained that empirical methods reduce to a minimum, or eliminate, the role of inductive and deductive reasoning by which data and ideas are processed in the scientific method. Empirical methods rely on uncritical acceptance of the data as they are being collected. To solve this problem, we need Q&A processes driven by scientific criticism and not Q&A processes that enforce the rules that govern the very empirical methods used in the data collection process. ◼️ The quantitative data are then analysed primarily by mathematical methods, which make no distinction between cause and effect. A fitting example is the tendency to fit (pun intended) scaling laws to synthetic rock mass data showing strength on the y-axis and model dimension or volumetric fracture intensity on the x-axis to justify size effects and representative elementary volume. Similarly, we feel that adding a sufficient amount of data to a scatter plot can reduce the impact of outliers. In both cases, we then use the resulting equations to design structures in rock. To shield our design from the qualitative, subjective and variable nature of our quantitative answer, we add a safety factor. Whether the safety factor reflects our confidence in the quantitative answer or our lack thereof is open to debate. ◼️ Although expressed using numbers and equations, a design process governed by empirical methods remains qualitative. Our empirical methods may be practical and logical, but we should not call them scientific.

THE 18KW HERITAGE TRAP: WHY EVIDENCE-BASESD ENGINEERING TRUMPS GENERIC MODELLING Heritage retrofit projects often stall because theoretical models fail to reflect reality. This pre-1919 Edinburgh semi-detached home was deemed a high risk for electrification when standard reports predicted a massive 18 kW heat loss. Such figures usually demand expensive power upgrades that kill the project before it starts. Barry Sharpe and Richard Hailstones of Renewable HEAT took a different approach. As members of the Guild of Master Heat Engineers, they applied empirical evidence over generic assumptions. By utilising research from Historic Scotland and Loco Home Retrofit, the team challenged standard U-values and air change rates. The result? The actual peak heat loss was only 10 kW. A year of monitoring via OpenEnergyMonitor confirms a Seasonal COP of 3.8. By integrating a heat pump with solar PV and battery storage, the household achieved a 44% reduction in total power bills. This case study proves that intelligent engineering can make heat pumps the most cost-effective solution even for solid stone properties. For the full technical breakdown and more evidence-based engineering insights, join the community at betateach.co.uk.

To become a top data analyst you need to be a strong problem solver! Follow this structure to find the real reasons behind business problems: 1. 𝗗𝗲𝗳𝗶𝗻𝗲 𝘁𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: Start by clearly stating the issue. For example, “We’ve observed a significant decrease in sales in the UK over the last few days.”   2. 𝗚𝗮𝘁𝗵𝗲𝗿 𝗗𝗮𝘁𝗮: Collect relevant information such as order processing times, customer service interactions, inventory levels, and active marketing campaigns.   3. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮: Use tools like SQL, Python, or Excel to analyze the data. Look for patterns, trends, and anomalies that could point to the root cause.   4. 𝗜𝗱𝗲𝗻𝘁𝗶𝗳𝘆 𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗖𝗮𝘂𝘀𝗲𝘀: Brainstorm all possible reasons for the issue. Use methods like the 5 Whys technique to investigate each potential cause more deeply.   5. 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗲 𝗛𝘆𝗽𝗼𝘁𝗵𝗲𝘀𝗲𝘀: Test your hypotheses against the data to see if they are supported. If not, refine your hypotheses and test again.   6. 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀: Once you’ve identified the root cause, support the business by showing possible solutions to address it. Monitor the results to ensure the issue is resolved. 𝗔 𝗿𝗲𝗮𝗹-𝘄𝗼𝗿𝗹𝗱 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗳𝗿𝗼𝗺 𝗺𝘆 𝗽𝗮𝘀𝘁: We notice an increase in customer lead time and here’s how we tackle it. 1. 𝗗𝗲𝗳𝗶𝗻𝗲 𝘁𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: “Customer lead time has increased by 20% in the last three months.”     2. 𝗚𝗮𝘁𝗵𝗲𝗿 𝗗𝗮𝘁𝗮: We collected data on order processing, sales forecast deviation, and shipping times.     3. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮: We found that the actual sales were in line with the forecast, and shipping times had remained constant. However, order processing times had increased significantly.     4. 𝗜𝗱𝗲𝗻𝘁𝗶𝗳𝘆 𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗖𝗮𝘂𝘀𝗲𝘀: We checked factors such as outages in warehouses, staffing issues due to high sickness rates, and process inefficiencies resulting from operating close to maximum capacity.     5. 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗲 𝗛𝘆𝗽𝗼𝘁𝗵𝗲𝘀𝗲𝘀: Data revealed that a spike in the sickness rate had reduced the available workforce.     6. 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀: We proposed to increase capacity buffers by 5% to 10% during the winter and hiring additional temporary workers to address the situation in the short term.   Following this approach for your root-cause analysis, you will become a valued problem-solving partner for your stakeholders. How do you ensure you’re addressing the root cause of an issue and not just the symptoms? ---------------- ♻️ 𝗦𝗵𝗮𝗿𝗲 if you find this post useful. ➕ 𝗙𝗼𝗹𝗹𝗼𝘄 for more daily insights on how to grow your career in the data field. #dataanalytics #datascience #rootcauseanalysis #problemsolving #careergrowth

How to read a PVsyst report (even if you're not an engineer) A PVsyst report can decide the future of a 10M project — but only if you know how to read it. You don’t need to be an engineer to understand whether a solar project makes sense. At the end of the day, a PV system has one job: produce reliable energy. Here’s how to quickly validate a PVsyst report like a pro: 1) Energy Yield (kWh/kWp) Is it realistic for the project location? Always benchmark against known regional values. If it looks too good to be true — it probably is. 2) DC/AC Ratio Is it within the typical range (1.1 – 1.25)? Too high → clipping losses. Too low → underutilized inverter capacity. This ratio directly impacts both performance and ROI. 3) Losses Breakdown Where is the biggest loss coming from? (Soiling, temperature, mismatch, wiring, etc.) Compare with benchmarks and question anything unusual. 👉 This is where real optimization happens. 4) Solar Irradiation Data Check monthly radiation values. Do they align with trusted databases (Meteonorm, NASA, Solargis)? Bad input = misleading output. 5) Project Inputs Location, system size, tilt, orientation, components… Small input errors = massive financial impact. Always validate assumptions before trusting results. Bottom line: A PVsyst report is not just a simulation… It’s a financial decision tool. If you can read it properly, you can spot risks, optimize design, and protect millions. If you're in solar development, investment, or EPC — this skill is not optional anymore.

Cloud computing infrastructure costs represent a significant portion of expenditure for many tech companies, making it crucial to optimize efficiency to enhance the bottom line. This blog, written by the Data Team from HelloFresh, shares their journey toward optimizing their cloud computing services through a data-driven approach. The journey can be broken down into the following steps: -- Problem Identification: The team noticed a significant cost disparity, with one cluster incurring more than five times the expenses compared to the second-largest cost contributor. This discrepancy raised concerns about cost efficiency. -- In-Depth Analysis: The team delved deeper and pinpointed a specific service in Grafana (an operational dashboard) as the primary culprit. This service required frequent refreshes around the clock to support operational needs. Upon closer inspection, it became apparent that most of these queries were relatively small in size. -- Proposed Resolution: Recognizing the need to strike a balance between reducing warehouse size and minimizing the impact on business operations, the team developed a testing package in Python to simulate real-world scenarios to evaluate the business impact of varying warehouse sizes -- Outcome: Ultimately, insights suggested a clear action: downsizing the warehouse from "medium" to "small." This led to a 30% reduction in costs for the outlier warehouse, with minimal disruption to business operations. Quick Takeaway: In today's business landscape, decision-making often involves trade-offs.  By embracing a data-driven approach, organizations can navigate these trade-offs with greater efficiency and efficacy, ultimately fostering improved business outcomes. #analytics #insights #datadriven #decisionmaking #datascience #infrastructure #optimization https://lnkd.in/gubswv8k

𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐓𝐡𝐫𝐨𝐮𝐠𝐡 𝐂𝐡𝐚𝐫𝐭𝐬: 𝐀 𝐆𝐮𝐢𝐝𝐞 𝐭𝐨 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 Data visualization is a powerful tool for 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 and 𝐜𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐧𝐠 insights from data. Different types of charts serve different purposes. Let's explore some common types of charts and their applications: 1️⃣ 𝐁𝐚𝐫 𝐂𝐡𝐚𝐫𝐭 📊: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Comparing categorical data or showing changes over time. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Comparing values of different categories, such as sales by product category or revenue by month. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Bar chart comparing monthly sales for different products. 2️⃣ 𝐋𝐢𝐧𝐞 𝐂𝐡𝐚𝐫𝐭 📈: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Showing trends and changes over time. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Visualizing continuous data over a period, such as stock prices over months or temperature variations over days. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Line chart showing the trend of website traffic over a year. 3️⃣ 𝐏𝐢𝐞 𝐂𝐡𝐚𝐫𝐭 🥧: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Displaying parts of a whole and illustrating proportions. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Showing the composition of a categorical variable, such as market share by product or distribution of expenses by category. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Pie chart illustrating the distribution of budget allocation for different departments. 4️⃣ 𝐇𝐢𝐬𝐭𝐨𝐠𝐫𝐚𝐦 📊: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Representing the distribution of continuous data. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Visualizing the frequency distribution of numerical data, such as age distribution of survey respondents or distribution of exam scores. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Histogram showing the distribution of heights among a sample population. 5️⃣ 𝐒𝐜𝐚𝐭𝐭𝐞𝐫 𝐏𝐥𝐨𝐭 📈: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Examining relationships between two continuous variables. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Identifying patterns and correlations between variables, such as the relationship between temperature and ice cream sales or the correlation between advertising spending and sales revenue. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Scatter plot depicting the relationship between hours studied and exam scores for students. 6️⃣ 𝐁𝐨𝐱 𝐏𝐥𝐨𝐭 📊: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Summarizing the distribution of numerical data and identifying outliers. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Visualizing the spread and skewness of data, comparing distributions, and identifying anomalies. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Box plot comparing the distribution of salaries for different job roles within a company. 7️⃣ 𝐇𝐞𝐚𝐭𝐦𝐚𝐩 🔥: 𝐏𝐮𝐫𝐩𝐨𝐬𝐞: Displaying the magnitude of a variable in a matrix format. 𝐒𝐮𝐢𝐭𝐚𝐛𝐥𝐞 𝐟𝐨𝐫: Visualizing relationships and patterns in large datasets, such as correlation matrices or user engagement matrices. 𝐄𝐱𝐚𝐦𝐩𝐥𝐞: Heatmap showing customer engagement levels across different demographics and products. Remember to choose the appropriate chart type based on the nature of your data and the insights you want to convey. #dataanalysis #visualization #charts #insights #analysis #eda Follow Sneha Vijaykumar for more... 😊