Re: [manrs-community] Upcoming changes to the MANRS measurement framework

Dear MANRS participants, We are glad to inform you that the features we announced back in December are now deployed in the MANRS Observatory. We also updated the Measurement framework https://www.manrs.org/manrs-observatory/measurement-framework/ to reflect these changes. Again, we encourage you to use the *feedback function* in the detailed reports to remove possible false positives and improve your MANRS readiness score. Best regards, • • • • *Pablo López-Aguilar* *Operations and Business Associate Director, Internet Integrity Program* Global Cyber Alliance plopezaguilar@globalcyberalliance.org +34 670527760 [image: GCA Website] <https://www.globalcyberalliance.org/> [image: GCA Twitter] <https://twitter.com/GlobalCyberAlln> [image: GCA Facebook] <https://www.facebook.com/GlobalCyberAlliance/> [image: GCA LinkedIn] <https://www.linkedin.com/company/global-cyber-alliance/> [image: GCA YouTube] <https://www.youtube.com/channel/UCC5x4cUnZqWsV7OqBx1vCOQ> [image: GCA GitHub] <https://github.com/GlobalCyberAlliance/> [image: GCA Email Signup] <https://www.globalcyberalliance.org/#signup> [image: GCA Instagram] <https://www.instagram.com/globalcyberalliance/> [image: GCA Forums] <https://community.globalcyberalliance.org/> GCA takes your privacy seriously. To review our privacy policy, please click here <https://www.globalcyberalliance.org/privacy-policy/>. On 14/12/2023, 15:03, "Andrei Robachevsky" <robachevsky@isoc.org> wrote: Dear MANRS participants, In July 2023, we started publishing the MANRS Readiness Scores <https://www.manrs.org/2023/06/manrs-readiness-scores-to-be-made-public-for-n...> for Actions 2, 3, and 4 (IRR and RPKI) for participants in the NetOps program. We didn't publish Action 1 scores at the time due to legitimate concerns within the community that the data used to calculate the scores produced too many false positives. As I presented during the MANRS Community Meeting on 28 November, we've done several improvements in the quality of these data and the scores. Below, you will find a summary of the upcoming changes — more detailed information can be found here: https://www.manrs.org/2023/12/the-road-to-publishing-action-1-netops-scores/. Please review these changes and let us know your questions or concerns. We plan to deploy these changes in January 2024. Note: Action 1 scores *will not be published yet* on the MANRS website to allow participants ample time to assess the changes' actual impact on the readiness scores. 1. The M1 and M1C metrics will no longer be included in the Action 1 score calculation. 2. The M2 and M2C metrics derived from BGPStream data have been removed completely from the Observatory due to low confidence. 3. The M2 and M2C metrics derived from GRIP data will be filtered based on the RPKI and IRR data to remove many false positives. 4. The M3 and M3C metrics will also be filtered. We will split all bogon prefixes into "full" and "administrative" bogons. The reason for doing so is that the operational practice is that "administrative bogons" are still being routed in some cases. Only "full" bogons will be considered when calculating the scores. 5. The M4 and M4C metrics will be filtered into "full" and "administrative bogons in the same way. These changes will lead to higher confidence in the data and the scores. Statistical analysis of filtered data revealed no false positives and, in some cases, led to an 80% reduction in reported incidents. We understand that false positives cannot be eliminated completely. To give you an opportunity to flag residual false positives a “feedback” function <https://www.manrs.org/2021/07/new-data-source-feedback-loop-enhance-manrs-ob...> has been in place since July 2021. After being reviewed by an auditor, such incidents will be removed. We encourage you to use *this function when reviewing your report*. We always welcome feedback on the MANRS Observatory and any of its data, whether due to these changes or for any other reason. You can send your feedback to this mailing list, email contact@manrs.org, or submit feedback via the website <https://www.manrs.org/about/contact/>. Regards, Andrei Robachevsky MANRS Secretariat
participants (1)
-
Pablo Lopez-Aguilar Beltran