MANRS Readiness Scores to be made public
Dear MANRS participants, While this only concerns the Network Operators program, this is an important milestone for the whole initiative. This change has been requested and discussed with the community with positive feedback. The MANRS Steering Committee has reviewed and approved this plan. Since April 2022, the MANRS team have been providing MANRS participants enrolled in the Network Operator program with an individual, monthly conformance report to update them on any changes to their compliance to MANRS Actions. This July, the scores for Actions 2, 3, and 4 will be made public on the MANRS NetOps participants table (https://www.manrs.org/netops/participants/), replacing the tick marks next to the named Actions. We will be adding the MANRS Readiness Score for Action 1 at a later date. This change aligns with the community’s objectives to: * Raise awareness of routing security problems and encourage the implementation of actions that can address them. * Improve transparency and credibility of the MANRS initiative. * Promote a culture of collective responsibility toward the security and resilience of the Internet’s global routing system. Network Operator participants will continue to receive individual, monthly conformance reports with their scores and any changes to the scores since the previous report at the start of each month. These scores will be updated on the MANRS website on the 8th day of each month. You can read more about how the MANRS Readiness Scores are calculated and their meaning<https://observatory.manrs.org/#/about> in the description of the Measurement Framework. Should you have any questions, please let us know. I also included a short Q&A to address the most common questions. Regards, Andrei Robachevsky MANRS Secretariat Questions & Answers Q: What is the MANRS Readiness Score? A MANRS readiness score indicates the degree of conformance of a MANRS participant to a MANRS Action. For example, to measure what degree Filtering (Action 1) is implemented we count the number of routing incidents where the network was implicated and the duration. Based on these metrics a normalized score is calculated for Action 1 for a specified time. Learn more about how the MANRS Readiness Scores are calculated and their meaning<https://observatory.manrs.org/#/about> in the description of the Measurement Framework. Q: Where will the MANRS Readiness Scores be made publicly available? * On the MANRS website, the scores will replace the tick marks in the participants' table<https://www.manrs.org/netops/participants/> of the Network Operators program. * On the MANRS Observatory, the scores will continue to be available to participants via their account. Q: Whose MANRS Readiness Scores will be made public? MANRS participants who are a part of the Network Operators program will be the first to have their MANRS Readiness Scores made publicly available. These will replace the tick marks for Actions 2, 3, and 4. MANRS Readiness Scores for Action 1 will be shown at a later date. Q: How do these MANRS Readiness Scores differ from the scores I get in my monthly conformance report? These scores are the same. This new initiative is making these scores public to align with the MANRS objectives to promote a culture of collective responsibility toward the security and resilience of the Internet’s global routing system and raise awareness of routing security problems and encourage the implementation of actions that can address them. Q: Why are the scores being made public? Publication of the MANRS Readiness Scores is an important step to improve credibility and transparency of the initiative. This has been requested by community members and the concept of making the MANRS Readiness Scores public was first proposed in 2021. Several steps have been made to improve the value of these scores, including adding these scores to the monthly conformance reports, validating incidents, and improving the quality of the data we use. We feel it’s at a point now that the scores provide an accurate enough analysis of the compliance of a MANRS participant towards MANRS Actions 2, 3, and 4. We will continue to work to refine the accuracy of the score for Action 1 before making this public. Q: How often will the scores be updated? You will continue to receive your most up-to-date MANRS Readiness Scores at the beginning of every month. These scores will then be published on the MANRS website on the 8th day of every month.
Hi, Please find my comments/questions below. Thanks in advance for your replies. Best regards, JMC. Orange Restricted De : Manrs-community <manrs-community-bounces@elists.manrs.org> De la part de Andrei Robachevsky Envoyé : jeudi 8 juin 2023 10:01 À : Community MANRS <manrs-community@elists.manrs.org> Cc : manrs-steering-committee@elists.manrs.org Objet : [manrs-community] MANRS Readiness Scores to be made public Dear MANRS participants, While this only concerns the Network Operators program, this is an important milestone for the whole initiative. <JMC> Why only Network Operators program? IMHO, CDN and IXP SHOULD be also considered if such a milestone is decided: these 2 programs have the "same maturity" as the Network Operator program. </JMC> This change has been requested and discussed with the community with positive feedback. <JMC> When? No log of such a discussion ... did I miss something? If this is the MANRS meeting during RIPE meeting, it was not be possible to attend it remotely ... and no report of this meeting too. </JMC> The MANRS Steering Committee has reviewed and approved this plan. Since April 2022, the MANRS team have been providing MANRS participants enrolled in the Network Operator program with an individual, monthly conformance report to update them on any changes to their compliance to MANRS Actions. <JMC> Seems there are still issues with MANRS initiative management before publishing data: * A questionnaire regarding one of my company's subsidiary has been sent a long time ago ... but no reply * A ticket has been submitted to get access to a subsidiary's account ... but no reply </JMC> This July, the scores for Actions 2, 3, and 4 will be made public on the MANRS NetOps participants table (https://www.manrs.org/netops/participants/), replacing the tick marks next to the named Actions. We will be adding the MANRS Readiness Score for Action 1 at a later date. This change aligns with the community's objectives to: * Raise awareness of routing security problems and encourage the implementation of actions that can address them. * Improve transparency and credibility of the MANRS initiative. * Promote a culture of collective responsibility toward the security and resilience of the Internet's global routing system. Network Operator participants will continue to receive individual, monthly conformance reports with their scores and any changes to the scores since the previous report at the start of each month. These scores will be updated on the MANRS website on the 8th day of each month. You can read more about how the MANRS Readiness Scores are calculated and their meaning<https://observatory.manrs.org/#/about> in the description of the Measurement Framework. Should you have any questions, please let us know. I also included a short Q&A to address the most common questions. Regards, Andrei Robachevsky MANRS Secretariat Questions & Answers Q: What is the MANRS Readiness Score? A MANRS readiness score indicates the degree of conformance of a MANRS participant to a MANRS Action. For example, to measure what degree Filtering (Action 1) is implemented we count the number of routing incidents where the network was implicated and the duration. Based on these metrics a normalized score is calculated for Action 1 for a specified time. Learn more about how the MANRS Readiness Scores are calculated and their meaning<https://observatory.manrs.org/#/about> in the description of the Measurement Framework. Q: Where will the MANRS Readiness Scores be made publicly available? * On the MANRS website, the scores will replace the tick marks in the participants' table<https://www.manrs.org/netops/participants/> of the Network Operators program. * On the MANRS Observatory, the scores will continue to be available to participants via their account. Q: Whose MANRS Readiness Scores will be made public? MANRS participants who are a part of the Network Operators program will be the first to have their MANRS Readiness Scores made publicly available. These will replace the tick marks for Actions 2, 3, and 4. MANRS Readiness Scores for Action 1 will be shown at a later date. Q: How do these MANRS Readiness Scores differ from the scores I get in my monthly conformance report? These scores are the same. This new initiative is making these scores public to align with the MANRS objectives to promote a culture of collective responsibility toward the security and resilience of the Internet's global routing system and raise awareness of routing security problems and encourage the implementation of actions that can address them. Q: Why are the scores being made public? Publication of the MANRS Readiness Scores is an important step to improve credibility and transparency of the initiative. This has been requested by community members and the concept of making the MANRS Readiness Scores public was first proposed in 2021. Several steps have been made to improve the value of these scores, including adding these scores to the monthly conformance reports, validating incidents, and improving the quality of the data we use. We feel it's at a point now that the scores provide an accurate enough analysis of the compliance of a MANRS participant towards MANRS Actions 2, 3, and 4. We will continue to work to refine the accuracy of the score for Action 1 before making this public. Q: How often will the scores be updated? You will continue to receive your most up-to-date MANRS Readiness Scores at the beginning of every month. These scores will then be published on the MANRS website on the 8th day of every month. ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Hi Jean-Michel,
Why only Network Operators program?
IMHO, CDN and IXP SHOULD be also considered if such a milestone is decided: these 2 programs have the “same maturity” as the Network Operator program.
The Network Operators programme has tangible and regularly updated readiness scores for each of its Actions, and the Observatory (from where this data is generated) is primarily based on this programme. The IXP and Vendors programmes have different Actions and do not currently have readiness scores to determine conformance because they’re largely about implementation of technical and organisational policy. On the CDN/Cloud programme this is a good suggestion and we can look at rolling this out next.
When? No log of such a discussion … did I miss something?
If this is the MANRS meeting during RIPE meeting, it was not be possible to attend it remotely … and no report of this meeting too.
It has been discussed in various community meetings (e.g. at RIPE and APRICOT) and by the Steering Committee over the last year or so. There has been significant support for doing this and really no objections to publishing data that can already be found the public domain. Do you have any specific concerns with publishing this data?
Seems there are still issues with MANRS initiative management before publishing data:
A questionnaire regarding one of my company’s subsidiary has been sent a long time ago … but no reply A ticket has been submitted to get access to a subsidiary’s account … but no reply Okay, apologies for this. We’re looking back at the tickets and will contact you off-list.
Regards, Kevin
Hi Kevin, At first, my apologies for this (very) delayed reply. Comments below. Thanks in advance for your reply. Best regards, JMC. Orange Restricted De : Kevin Meynell <meynell@isoc.org> Envoyé : jeudi 8 juin 2023 11:44 À : COMBES Jean-Michel INNOV/NET <jeanmichel.combes@orange.com> Cc : Andrei Robachevsky <robachevsky@isoc.org>; Community MANRS <manrs-community@elists.manrs.org>; manrs-steering-committee@elists.manrs.org Objet : Re: [manrs-community] MANRS Readiness Scores to be made public Hi Jean-Michel, Why only Network Operators program? IMHO, CDN and IXP SHOULD be also considered if such a milestone is decided: these 2 programs have the "same maturity" as the Network Operator program. The Network Operators programme has tangible and regularly updated readiness scores for each of its Actions, and the Observatory (from where this data is generated) is primarily based on this programme. <JMC> OK. Even if, I partially disagree with "tangible" (cf. below about the measurement framework). </JMC> The IXP and Vendors programmes have different Actions and do not currently have readiness scores to determine conformance because they're largely about implementation of technical and organisational policy. <JMC> OK. I was not aware for IXPs. </JMC> On the CDN/Cloud programme this is a good suggestion and we can look at rolling this out next. <JMC> OK. </JMC> When? No log of such a discussion ... did I miss something? If this is the MANRS meeting during RIPE meeting, it was not be possible to attend it remotely ... and no report of this meeting too. It has been discussed in various community meetings (e.g. at RIPE and APRICOT) and by the Steering Committee over the last year or so. There has been significant support for doing this and really no objections to publishing data that can already be found the public domain. <JMC> IMO, such a decision should have been discussed/voted during a MANRS community meeting (cf. https://www.manrs.org/about/governance/community-charter/). Indeed, everyone is not able to attend physically side meetings during RIPE/APRICOT/etc. meeting and, as mentioned (cf. Andrei's email), this publication is an important step for the MANRS initiative. BTW, "data that can already be found the public domain" is not correct from my point of view: this is a publication of a public data *interpretation*, based on the measurement framework *specified by the MANRS initiative*. Unfortunately, this one is not perfect. Especially, for network operators having a large customers cone (e.g., Tier-1), as I raised during the specification of this framework. Maybe, it should be wise/useful to provide a "warning" message on the WWW site (e.g., "MANRS readiness index is measured per Action using this measurement framework, which doesn't take into account network operator's customers cone - potential new version of this framework could solve this issue")? </JMC> Do you have any specific concerns with publishing this data? <JMC> Yes - based on the RIPE side meeting slides, about: * Process - OPT-IN by default Generally, OPT-OUT is not well appreciated (i.e., looks like an "enforced" action) ... I would have preferred a "OPT-IN" (i.e., OPT-OUT by default) based process. BTW, must an operator OPT-OUT each month if it wants to be OPT-OUTed? * Process - Delay to review IMO, 15-days review would be wiser. * Data - Measurement Framework Cf. previous comment on measurement framework and customers cone impacting the operator's scores. </JMC> Seems there are still issues with MANRS initiative management before publishing data: * A questionnaire regarding one of my company's subsidiary has been sent a long time ago ... but no reply * A ticket has been submitted to get access to a subsidiary's account ... but no reply Okay, apologies for this. We're looking back at the tickets and will contact you off-list. Regards, Kevin ____________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Hi Jean-Michel, Kevin is on vacation, let me jump in. On 30/06/2023, 17:31, "jeanmichel.combes@orange.com" <jeanmichel.combes@orange.com> wrote: […] It has been discussed in various community meetings (e.g. at RIPE and APRICOT) and by the Steering Committee over the last year or so. There has been significant support for doing this and really no objections to publishing data that can already be found the public domain. <JMC> IMO, such a decision should have been discussed/voted during a MANRS community meeting (cf. https://www.manrs.org/about/governance/community-charter/). Indeed, everyone is not able to attend physically side meetings during RIPE/APRICOT/etc. meeting and, as mentioned (cf. Andrei’s email), this publication is an important step for the MANRS initiative. Yes, good point. We will make sure our communication is better next time. BTW, “data that can already be found the public domain” is not correct from my point of view: this is a publication of a public data *interpretation*, based on the measurement framework *specified by the MANRS initiative*. Unfortunately, this one is not perfect. Especially, for network operators having a large customers cone (e.g., Tier-1), as I raised during the specification of this framework. Maybe, it should be wise/useful to provide a “warning” message on the WWW site (e.g., “MANRS readiness index is measured per Action using this measurement framework, which doesn’t take into account network operator’s customers cone – potential new version of this framework could solve this issue”)? Could you please elaborate on the problems related to the customer cone? </JMC> Do you have any specific concerns with publishing this data? <JMC> Yes – based on the RIPE side meeting slides, about: - Process – OPT-IN by default Generally, OPT-OUT is not well appreciated (i.e., looks like an “enforced” action) … I would have preferred a “OPT-IN” (i.e., OPT-OUT by default) based process. BTW, must an operator OPT-OUT each month if it wants to be OPT-OUTed? No, one request is enough. If someone wants to be opted-out they should contact the secretariat at contact@manrs.org<mailto:contact@manrs.org>. For those who oped-out tick marks will be shown instead of the scores. - Process – Delay to review IMO, 15-days review would be wiser. - Data – Measurement Framework Cf. previous comment on measurement framework and customers cone impacting the operator’s scores. </JMC> I appreciate there are parameters we as the community would like to tune. That is why we rolled this out in two phases. Let us run it for some time and review the situation before rolling out the next phase. Answers to the questions like you ask should be more clear then. Thank you for your feedback, Regards, Andrei
participants (3)
-
Andrei Robachevsky -
jeanmichel.combes＠orange.com -
Kevin Meynell