MANRS Community

Download

manrs-community@elists.manrs.org

December 2023

1 participants
1 discussions

Upcoming changes to the MANRS measurement framework
by Andrei Robachevsky Dec. 14, 2023

Dec. 14, 2023

Dear MANRS participants, In July 2023, we started publishing the MANRS Readiness Scores<https://www.manrs.org/2023/06/manrs-readiness-scores-to-be-made-public-for-…> for Actions 2, 3, and 4 (IRR and RPKI) for participants in the NetOps program. We didn't publish Action 1 scores at the time due to legitimate concerns within the community that the data used to calculate the scores produced too many false positives. As I presented during the MANRS Community Meeting on 28 November, we've done several improvements in the quality of these data and the scores. Below, you will find a summary of the upcoming changes — more detailed information can be found here: https://www.manrs.org/2023/12/the-road-to-publishing-action-1-netops-scores/. Please review these changes and let us know your questions or concerns. We plan to deploy these changes in January 2024. Note: Action 1 scores will not be published yet on the MANRS website to allow participants ample time to assess the changes' actual impact on the readiness scores. 1. The M1 and M1C metrics will no longer be included in the Action 1 score calculation. 2. The M2 and M2C metrics derived from BGPStream data have been removed completely from the Observatory due to low confidence. 3. The M2 and M2C metrics derived from GRIP data will be filtered based on the RPKI and IRR data to remove many false positives. 4. The M3 and M3C metrics will also be filtered. We will split all bogon prefixes into "full" and "administrative" bogons. The reason for doing so is that the operational practice is that "administrative bogons" are still being routed in some cases. Only "full" bogons will be considered when calculating the scores. 5. The M4 and M4C metrics will be filtered into "full" and "administrative bogons in the same way. These changes will lead to higher confidence in the data and the scores. Statistical analysis of filtered data revealed no false positives and, in some cases, led to an 80% reduction in reported incidents. We understand that false positives cannot be eliminated completely. To give you an opportunity to flag residual false positives a “feedback” function<https://www.manrs.org/2021/07/new-data-source-feedback-loop-enhance-manrs-o…> has been in place since July 2021. After being reviewed by an auditor, such incidents will be removed. We encourage you to use this function when reviewing your report. We always welcome feedback on the MANRS Observatory and any of its data, whether due to these changes or for any other reason. You can send your feedback to this mailing list, email contact(a)manrs.org<mailto:contact@manrs.org>, or submit feedback via the website<https://www.manrs.org/about/contact/>. Regards, Andrei Robachevsky MANRS Secretariat

1 0