Adding Metrics section to capabilities in understanding domain

[KevDLR]

🛠️ Description

Adding a Metrics/ KPI section to the capability to provide guidance on the Metric lens of the FinOps assessment.

Fixes
N/A

📋 Checklist

🔬 How did you test this change?

• 🤏 Lint tests
• 🤞 PS -WhatIf / az validate
• 👍 Manually deployed + verified
• 💪 Unit tests
• 🙌 Integration tests

🙋‍♀️ Do any of the following that apply?

• 🚨 This is a breaking change.
• 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

• ✅ Updated changelog (required for dev PRs)
• ➡️ Will add log in a future PR (feature branch PRs only)
• ❎ Log not needed (small/internal change)

📖 Did you update documentation?

• ✅ Public docs in docs (required for dev)
• ✅ Internal dev docs in src (required for dev)
• ➡️ Will add docs in a future PR (feature branch PRs only)
• ❎ Docs not needed (small/internal change)

[flanakin]

This looks good. I haven't looked at it from a completeness perspective, but I tink it's a great list! My comments are mostly around landing the right way to bring metrics into the guide altogether.

[flanakin]

Let's use the same headers across all files so we can link to them generically. Also note we should use sentence casing rather than title casing to align with the Microsoft Style Guide.

Suggested change

	## Data Ingestion Metrics
	## KPIs and metrics

[flanakin]

Update all of the categories to be sentence cased to align to the Microsoft Style Guide.

[flanakin]

Each KPI should include a formula. We may not be able to format this as a table.

[flanakin]

How feasible is it to measure this? I'm not pushing back. It sounds like the right thing to do, but do they have a way to actually measure it? How would we calculate this for them? Should we outline any potential challenges they may have in collecting this to give them a heads up? I'd hate for someone to take this list and say, "let's go track all these" and then realize there's no way to do it.

[flanakin]

Can you add each one of these into the backlog for adding to Power BI?

[flanakin]

I like this one. A few thoughts:

1. Do we need to call out that latency may differ by dataset?
2. Do you intend to use "mean" time? Not average or percentile? All have merits, so just confirming.
3. This can likely be split into multiple KPIs.
4. Is latency trend a KPI or a visualization of a KPI over time? Not sure if visualizations need to be called out here unless we need to speak to the value of the visual. I'm open to either approach. Just thinking out loud to keep this simple. If we do keep it, it's probably a separate KPI that might be better if we can quantify a single number for it. Not sure 🤔

[KevDLR]

I have added per dataset in the Formula column, adjusted to average instead of mean time, and have removed the trend component as this is a result of the KPI

[flanakin]

This name needs some work, but I do like it. I've thought about this one as well. We need to know what data is missing so we can backfill it. Should this be bound to months with complete data over the retention/reporting period?

[KevDLR]

Maybe along the lines of "Historical data span"? (Other options include, Data lookback period, Data retention period
, Ingested data horizon, Data history range)

I think so because that'll ensure the KPI reflects the availability of fully usable historical data, providing a more accurate measure of data completeness and reliability over time. How do you suggest I modify?

[flanakin]

This brings up a question about how much people are using historical data. We should probably talk about the cost of each month of data compared to the usage of that data. If people aren't using it, then that's wasted money. That will also help them quantify the value of storing the historical data.

[flanakin]

Can you think about the cost and carbon impact of each one of these? It may not apply everywhere. Anything that comes back to something that is metered, like data size or compute time.

[flanakin]

Similar comments about trends on this one. It's very interesting. This warrants its own backlog to think thru whether we have the right guidance to support it.

[KevDLR]

@KevDLR please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

[flanakin]

I like where this is going. The main issue I see is that the formulas aren't formulas and we might need more context as to why they should care about tracking each KPI.

[flanakin]

Make sure the formulas are actual formulas. This applies to all KPIs.

Also, avoid the use of "cloud". We need to remove that everywhere in the next Framework update so that will make the next update easier.

Suggested change

	| Cost allocated | Evaluates the extent to which cloud costs are allocated among organizational units.| Percentage of cloud cost allocated. |
	| Cost allocated | Evaluates the extent to which costs are allocated among organizational units. | {Allocated cost amount} / {Total cost} * 100 |

[flanakin]

What value is this supposed to be? Granularity sounds like an attribute and not a metric.

[flanakin]

nit: Do we need both this and allocated %? I'm not opposed to having both but is it useful to have 83% next to 17%? Seeing one is obvious what the other is. Not sure which is more important, tho. Maybe we want to describe both to let them choose. If that's the case, we might also want to include a justification column for why each metric is important. Thoughts?

[flanakin]

You don't need to bold a header. They're bolded based on the site styles.

Suggested change

	| **KPI** | **Definition** | **Formula** |
	| KPI | Definition | Formula |

[flanakin]

Is this already implicitly covered by the allocated %?

[flanakin]

I like what I think this is trying to cover: feedback on reports. If that's what it is, it probably needs to be reworded to be a little clearer. The KPI name should be a metric and the formula should be a formula. I'm wondering if there's a diference between new feedback in a period, resolved feedback in a period, and total active/unresolved feedback. We should also add this to the backlog as something for us to add to our reports.

[flanakin]

How would this be measured? Would DAU, WAU, and MAU be more achievable? I like the idea of tracking adoption, but that's just the first use and not the ongoing use. I wonder how many FinOps teams would really care to get into adoption, engagement, and retention. Although, the same could be asked about MAU. since it doesn't indicate success or give context on growth 🤔

[flanakin]

Wouldn't this belong in data ingestion?

[flanakin]

Data ingestion? How is this measured?

[flanakin]

How useful is this? Every report will have a different amount of complexity and, in theory, teams shouldn't be churning out new reports endlessly. I like the idea of it, but I suspect new reports would be rare once a team is established. They'll likely update existing ones more.