Initial design considerations to support new southbounds

[viniarck]

Initial design considerations to support new remote southbound APIs on Kytos-ng

Goals

• Get the discussion started for which southbound (SB) protocols need to be supported for use cases at AmLight in the next 2 to 3 years.

• Understand if what's needed fits well in the ecosystem and Kytos-ng architecture.

• Initial Kytos-ng architecture considerations to support the new SB(s).

• Initial understanding how the platform and NApps should evolve to support the new SB(s).

Current Southbound Introduction

• Kytos has been designed to be a single lightweight SDN OpenFlow controller with an ecosystem of network applications (NApps):

                 kytosd                    
    +-------------------------------------+
    |                                     |
    |                                     |
    |            PubSub event bus         |
    |     --+--^------------+--^-----     |
    |       |  |            |  |          |
    |  +----v--+----+   +---v--+------+   |
    |  |            |   |             |   |
    |  |  TCP Server|   |  of_core    |   |
    |  | (OpenFlow) |   |(with pyof)  |   |
    |  +-^--------^-+   |             |   |
    |    |        |     +-------------+   |
    |    |        |                       |
    |    |        |                       |
    |    |        |                       |
    +----+--------+-----------------------+
         |        |                        
         |        | (n OpenFlow 1.3 switches)
         |        |                        
+--------+-+    +-+--------+               
|          |    |          |               
|          |    |          |               
|   SW 1   |    |   SW N   |               
|          |    |          |               
|          |    |          |               
+----------+    +----------+               

• The core and NApps were designed to support OpenFlow, without overabstraction and overgeneralization, solid design choice back then that facilitated code maintenance.
• Any SB that exposes flow table entries management, ports and other general functions and their status via a TCP connection can fit well incrementally with the current architecture.
• Kytos-ng can evolve gradually to support other OpenFlow-ish like remote southbound protocols.

Southbound, Use Cases and Requirements

Which SB APIs can Kytos-ng try to support?

• This should be driven by the network use cases and requirements of the current main project sponsor: AmLight
• The SB APIs need to also be available on AmLight switches or be in the future roadmap where it's certain it will be available.
• Although the focus is to R&D and solve pratical network challenges at AmLight, it can also be used by other compatible programmable networks too (collaboration/partnership).
• Kytos-ng should evolve while leveraging its main strengths, depending on the goals, if it starts to diverge too much, then we need to reasess if Kytos-ng is still the right choice.

Previous network diagrams that Jeronimo has shared to think about new SBs:

+--------------------------------------------------------+       
|                                                        |       
|                  Kytos-ng                              |       
|                                                        |       
+--------------------------------------------------------+       
             |                           |            
+-----------------------+ +------------------------------+       
| of_core: OpenFlow 1.3 | |TBD API: BFRuntime|P4Runtime  |       
|                       | |SONiC API |Others             |       
+----^----------^-------+ +---^---------^-----------^----+       
     |          |             |         |           |            
     |          |             |         |           |            
+----+---+  +---+----+    +---+----+ +--+-----+ +---+----+       
|        |  |        |    |        | |        | |        |       
|  SW1   |  |  SW2   |    |  SW3   | |  SW4   | |  SW5   |       
|        |  |        |    |        | |        | |        |       
+--------+  +--------+    +--------+ +--------+ +--------+       

Main goals: eventually, phase out OpenFlow 1.3 and introduce new SBs

Topology

• Each Switch in the platform would need to be augmented to explicitly have which southbound it supports, OpenFlow being the default for now.
• Different southbounds, especially if the controller connects to them connecting to the controller, would need to be created (where its type/kind will be set).
• This southbound kind or capability will be used for filtering, the supported version is typically negotiated or is implicit in the protocol.
• Switch/Interface attrs would ideally still use a subset of the current abstractions normalized while allowing extra optional attrs when absolutely needed

Protocols and network applications terminology

This subsection will introduce a new terminology:

• Flow Network Application (FNApp): refers to a higher level network functionality that is flow-based, for instance, mef_eline.
• Infra Network Application (INApp): refers to a NApp that provides a network infracture functionality that's not related to flows, for instance pathfinder, topology, maintenance.

The following options are related to how a protocol or API can be integrated in a NApp (assuming that the platform still needs to work with OpenFlow + another SB(s)):

• Option SS (SINGLE protocol SINGLE choice): this is currently (2024.2) what all NApps are, they only support OpenFlow and OpenFlow is the only choice.
• Option SM (SINGLE protocol MULTIPLE choices): this is when the network resources flow/switches/links/interfaces will only deal with a single protocol but allow multiple choices. To give an example with mef_eline, Option SM would mean, an EVC could either be provisioned with flows of a southbound X or southbound Y.
• Option MM (MULTIPLE protocols MULTIPLE choices): this is when the network sources can be mixed with multiple types, and example with mef_eline would be a same EVC would use different southbounds (whether or not this is abstracted), and for pathfinder it would mean beaing able to find paths on devices with a single southbound kind or multiple different kinds.

The naming of the protocol options SS/SM/MM were defined from a network flow-based service point of view, when the NApp doesn't use flows, then you can consider in terms of other network resources it uses such as Switches/Interfaces/Links

By introducing another SB it unlocks network FNApps and INApps to potentially be:

• Option SM:
  • Examples:
    • EVC A from SW1 to SW2 using OpenFlow.
    • EVC B from SW3 to SW5 using a new SB.
  • It can fit well with the current architecture, although only introducing a new SB strategically and knowing the team has capacity to maintain.
• Option MM:
  • Superset evolution of the prior option, it can evolve gradually if there's a critical reason and use cases to justify building and maintain it.
  • A higher level NApp (when it makes sense) can act like a facade to two or more specific SB NApps.
    • This would be more composable when NApp-to-NApp controllers/managers calls are allowed (tech debt to be solved).
  • Examples:
    • EVC C from SW1 to SW5 using OpenFlow 1.3 on SW1 to SW2 and using new SB on SW3 to SW5 (notice EVC C is a single EVC, multiple protocols being used in the same EVC).

Options aren't mutually exclusive, different NApps can use different options, it'll mostly boil down to its SB dependency, but the order from complexity typically is: SS, SM and MM.

Challenges to be aware regarding Option MM:

• Not necessarily all matches and actions are available on every switch regardless of its southbound.
  • Plus, for FNApps, while OpenFlow is still around, this also implies in having to still support current applications encapsulation like stacked svlans per EVC, which might or might not be a good depending on AmLight future network resource optimizations goals.
• Trying to support different protocols with service stitching by breaking down into same protocols pairs would be extremely difficult to to maintain and understand its status, troubleshoot, debug both in terms of code and day to day network operations.
  • This an example where it diverges too much from Kytos-ng architecture and its strengths.

Notice that for FNApps, option MM is essentially a service agnostic orchestrator which might start to diverge too much from the current architecture, the next subsections will provide suggestions how it can be evolved in maintainable way while still being aligned with current major characteristics of the platform and the ecosystem.

Notice that for INApps (and NApps that has minimal SB dependency), option MM isn't too difficult to evolve or be augmented, for example, pathfinder, topology, maintenance, they all trying to support multiple protocols over multiple Switches/Interfaces/Links mostly comes down to using standardized attributes (which would be already well known), and filtering with multiple southbound types or invidiaual ones, and by default assuming OpenFlow not to breaking compatibility as of now. A table below will provide more information about the current NApps and their SB dependencies.

Which southbounds would need to be supported in the short and medium-term?

• Q1) Which physical/virtual switches models will be used and supported at AmLight for at least 3 years?
  • Answered at AHM: Physical ones: Tofino 1, Tofino 2, and maybe xsight x2; Virtual ones: tofino virtual model.
• Q2) Will FNApps with option MM (mef_eline using different southbounds in the same EVC) be needed in the next 3 years?
  • Answered at AHM: NOT mandatory, although desirable to keep in mind
• Q3) Would FNApps with option SM with ONE new SB be enough for the next 2 years? Which SB does AmLight need to use?
  • Example: OpenFlow + BFRuntime API or OpenFlow + P4Runtime API (also compatible with SONiC PINS + gNMI)?
  • Answered at AHM: Yes, one extra new SB is expected to suffice. Still TDB which one though.
• Q4) Would FNApps with option SM with TWO new SBs be enough for the next 3 years? Which SBs does AmLight need to use? Are they mandatory or only desirable? Is it really needed or can we stick with just one extra SB?
  • Example: OpenFlow + BFRuntime + P4Runtime API?
  • Answered at AHM: Likely not

Notice that not necessarily you need to translate between protocols to be able to use more than one SB in concurrently in the same kytosd instance. In fact, leveraging the strengths of kytos by loading specific option SS NApps or open SM NApps you users can load the set of NApps that are needed to support certain southbounds. This is mostly an implementation detail, and common parts can eventually emerge with option MM when justifiable with major gains. kytosd is a single process running in a Python runtime, what a NApp is responsible for and what it does in its modules is merely a logical software organization for typically focusing on doing one major thing and this major network thing well. Think of NApps as dynamic loadable plugins, which some provide lower-ish level network infrastructure functionality such as topology/pathfinder/of_core while other provide higher level network flows services such as mef_eline, telemetry_int, sdntrace_cp, sdntrace.

Which problem(s) the new SB is supposed to solve at AmLight?

• Q5) Which problem(s) the new SB is supposed to solve at AmLight? and how AmLight expects to use the SB or SBs and their network use cases?
  • @jab1982 @italovalcy @RenataFrez and folks from the Ops team said they will consolidate a document with the network use cases and how the topology would look like (we also spoke about contiguous adjacency same protocol and also non contiguous) epic: create a roadmap with future use cases to be supported by Kytos #526

To get the discussion started:

• a) Replace OpenFlow and OpenFlow NApps gradually due to EoL concerns?
  • a1) How would the roll out deployment and e2e tests look like?
  • a2) Start supporting another SB while phasing out OpenFlow 1.3
• b) Unlock new network applications where OpenFlow might not provide full data plane programmability?
  • b1) BFRuntime API
    • Vanguard of P4Runtime API (thanks to Tofino Native Architecture)
    • SDE toolchain:
      • Intel P4 Studio SDE is now open-source as of Jan 15, 2025 (Apache licesnsed, used to be Intel proprietary)
      • bfruntime.proto
      • What if we hit a major SDE related bug that is blocking? Will our team potentially try to fix it in the worst case? Open source community is a positive
    • Tofino stalled R&D business risks
  • b2) P4Runtime API
    • Should we try to also support P4Runtime API v1?
      • Is it current version sufficient for the network applications use cases? Network convergence / port status polling via other APIs?
      • Is it aligned with one of our major goals? Do we have physical switches supporting it? What else it can unlock? Be appealing to other users/researchers?
      • Can it be strategic for AmLight due to SONiC PINS?
    • Compared to BFRuntime API: different protobuff, fewer supported functionalities; has a similar high level goal as far as its southbound API; slow moving standard as of Dec 2024.
  • b3) Does AmLight need FGPA reconfigurable logic where P4 data plane programmability might not suffice for major use cases? How will it be managed?
• c) Will a single network service like an EVC still use a single protocol but with multiple choices or will it need to support multiple protocols in the same EVC?
• d) Try to reuse other open source control plane open source traditional protocols (such as SONiC/FRR) + integrate with SoNIC PINS (P4Runtime API) + gNMI, so only program the data plane via P4 where absolutely need to aggregate application value?
  • This approach also needs to refine how the legacy part would integrate to still create certain network resources that needs to be represented in the platform, for instance, if link discovery were to be done by LLDP via FRR, how would kytos integrate would it be poll or push / subscription based?
• e) any other remote OpenFlow-ish protocol/P4 Runtime API like?

a), b1), b2), and e) fit well with the current Kytos-ng architecture and would also align well with its evoluion as a remote SDN controller platform. option d) can also fit well if PINS + gNMI works well; c) is one of the major requirements that will shape large design decisions from this point forward.

All of these prior points, requirements and questions need to be answered, refined, and aligned with AmLight network engineering team use cases.

Current core libraries & NApps OpenFlow dependencies

• The tables below tries to quantify how much each project depends on OpenFlow (messages & data models) in a scale from none to full: none, low, medium, high, full.

Core Project	OpenFlow dependency
python-openflow	full
kytos	low+ (class data models and tests)
ui	low (API models)
kytos-utils	none

NApp	OpenFlow dependency	Proposed Evolution Option
kytos/of_core	full	SS
kytos/noviflow	full	SS
kytos/telemetry_int	full (due to NoviFlow OpenFlow experimenter messages dependency)	SS (SM only if P4 pipeline doesn't cover it)
kytos/flow_manager	high	SS (to optimize DB lookups)
amlight/sdntrace_cp	medium	SS (most likely)
kytos/of_multi_table	medium	SS
amlight/sdntrace	low	SM
kytos/of_lldp	low	SM
amlight/coloring	low	SM
kytos/mef_eline	low	SM
amlight/kytos_stats	low	MM (potentially normalized)
kytos/topology	low (entity SB type filters)	MM
kytos/pathfinder	low (entity SB type filters)	MM
kytos/sdx	low (entity SB type filters maybe)	MM
kytos/maintenance	none	MM

General New Patterns to support new SBs

If we were to hypothetically develop and have to port all NApps to another SB:

NApps will typically choose an option SS, SM or MM and stick with it. If the NApp has too much south bound specific dependency it should be SS (such as of_core), otherwise SM (such as mef_eline in the future), and if it's not too heavy flow-based, then MM.

Think of an Option SS NApp as a protocol adapter pattern at a NApp level, where as Option SM NApp has a protocol adapter pattern at its internal flow builder and handler methods, and option MM for INApps trying aggregating and/or potentially normalizing and filtering for the southbound/capability kinds.

The general idea is to still keep a NApp responsible for doing one thing (network functionality or responsibility) and one main thing well, and only support multiple protocol choices in the same NApp if it doesn't have too much protocol details dependency that were made in its design.

Analogy with aiop4 and p4_l2ls

• Another SS option NApp example: modularity-wise and connections-wise it's analogous to of_core + TCP server, except p4s_core does what of_core but for P4Runtime API.
  • p4s_core uses aiop4, which is responsible for exposing asyncio gRPC client and managing the protocol state machine and its messages
  • p4_l2ls is a P4Runtime example NApp that can perform L2 switch learning functionalities by subscribing to events and pushing P4 table entries

Note: As of Dec 2024, p4_l2ls is also doing part of p4s_core code responsibilities since this was an initial proof of concept

                 kytosd                    
    +-------------------------------------+
    |                                     |
    |                                     |
    |            PubSub event bus         |
    |     --+--^------------+--^-----     |
    |       |  |            |  |          |
    |  +----v--+-----+   +--v--+-------+  |
    |  |             |   |             |  |
    |  |  p4s_core   |   | p4_l2ls     |  |
    |  | (with aiop4)|   |             |  |
    |  +-------------+   |             |  |
    |    |        |      +-------------+  |
    |    |        |                       |
    |    |        |                       |
    |    |        |                       |
    +----+--------+-----------------------+
         |        |                        
         |        | (n P4Runtime API v1.4 switches)
         |        |                        
+--------v-+    +-v--------+               
|          |    |          |               
|          |    |          |               
|   SW 1   |    |   SW N   |               
|          |    |          |               
|          |    |          |               
+----------+    +----------+               

Hypothetical future new SB

• Now, if we had to support another new SB concurrently (for example, P4Runtime API):
• TCP Server + of_core + pyof for managing OpenFlow switches (in the meantime as OpenFlow gets phased out)
• p4s_core + aiop4 for managing P4Runtime API switches

                 kytosd                    
    +-------------------------------------------------------------+
    |                                                             |
    |                                                             |
    |            PubSub event bus                                 |
    |     --+--^------------+--^--------------+--^----            |
    |       |  |            |  |              |  |                |
    |  +----v--+-----+   +--v--+-------+  +---v--+------+         |
    |  |             |   |             |  |             |         |
    |  |  p4s_core   |   | TCP Server  |  |   of_core   |         |
    |  | (with aiop4)|   | (OpenFlow)  |  | (with pyof) |         |
    |  +-------------+   |             |  |             |         |
    |    |        |      +------^------+  +-------------+         |
    |    |        |             |                                 |
    |    |        |             |                                 |
    |    |        |             |                                 |
    +----+--------+-------------+---------------------------------+
         |        |             |           
         |        |             | 
         |        |             |          
+--------v-+    +-v--------+  +-^----------+
|          |    |          |  |            | 
|          |    |          |  |            | 
|   SW 1   |    |   SW 2   |  |   SW 3     | 
|  (P4 R.) |    |  (P4 R.) |  |(OpenFlow)  | 
|          |    |          |  |            | 
+----------+    +----------+  +------------+

• or p4bf_core + aiobf for managing BFRuntime API switches (this hasn't been prototyped yet, but it's expected to work based on prior training knowledge)

Related Notes

General new guidelines that will still be refined:

• Any medium+ dependent project that is being proposed as option SS is expected to be forked covering another SB, users can enable and load the set of NApps and compatible SBs they are looking for.
  • Think of it as the adapter software design pattern, except at a NApp level.
  • For NApps that has low OpenFlow dependency, like mef_eline its flow building methods will be refactored to be a flower builder (or factory) for a given SB/data model. For EVC creation, we'll need to discuss if option SM is good enough initially, and then only allow creation/updates of the same type.
  • Some NApps such as telemetry_int can also be entirely offloaded to the pipeline dataplane design.
    • All INT telemetry capabilities wouldn't require a NApp performing control plane actions by a NApp, INT could be always enabled if that's safe seamlessly.
    • This also minimizes effort to maintain certain network functionalities that today is a NApp in OpenFlow and Python land.
    • This is also a good example about how powerful and feature-rich a P4 pipeline can be when done right as far as network programmability.
• The NApps which will need to be recreated, certain agnostic major parts might get extracted as a new Python package or lib for the core team to maintain to be reused.
• For example, flow_manager consistency check, most of it is protocol flow entry agnostic, except for a few fields that can be parametrized.
• The biggest parts will be evident and will emerge when designing and planning the new NApp scope.
• NApps with major changes should also consider fixing certain tech debts along the way strategically.
• A higher level NApp (when option MM is needed) can act like a facade to two or more specific SB NApps.
  • This would be more composable when NApp-to-NApp controllers/managers calls are allowed (tech debt to be solved).
  • Option MM can be built from option SM composable NApps too.
• When adding support for a new SB, each NApp will need a blueprint or issue discussion to decide if the forks for each SB will be needed or not and make the Option selection official, and also document what needs to be augmented and changed, while also trying to maintain compatibility with OpenFlow while it's still around. For instance, pathfinder has low SB dependency, so it shouldn't need a fork, and all of its funcionality should still filter for OpenFlow only by default, but it should evolve to allow searching paths that's compatible with a given SB.
• flow_manager for instance would likely be option SS:
  • flow_manager (OpenFlow).
  • flow_manager_bfrt (if it were for BFRuntime).
  • flow_manager_p4rt (if it were for P4Runtime).
• Eventually flow_manager_p4 could also turned into option MM if justifiable (although it'd be difficult, it wouldn't be as difficult as NApps which are e2e higher level applications like mef_eline):
  -flow_manager_p4 could compose both flow_manager_bfrt and flow_manager_p4rt.
  • It also opens up opportunity for certain cases to also expose a common normalized or subset API when there are substantial major benefits to do so.
  • A MM NApp can also learn lessons from the more specific NApps, it's a gradual path where new abstractions if needed will emerge naturally without trying to force a wrong one.
  • Not all MM NApps need to compose multiple NApps, only the ones who are heavily dependent on southbound details.

How far is Kytos-ng ecosystem to support a new SB?

• Depends on the use cases requirements and goals

  • Which SB(s)
  • Which NApps (or which NApp functionalities)
  • New NApps
  • How to setup local development new SB with mininet
  • New unit/e2e tests supporting the new SB too
  • New well written blueprints how this should be built

• Many of these items can be done in parallel if the team has capacity

• However, the team currently is also responsible for maintaining Kytos-ng OpenFlow and NApps in production

• Ballpark figure: to get an equivalent OpenFlow Kytos-ng ecosystem but in P4 production-grade will likely take a few years with an experienced full-time small team, but do we need an entire equivalent NApp ecosystem? Will we still maintain the same NApps or try to also trim them down?

  • Assuming we don't encounter major issues with:
    • P4 pipeline match and actions for the expected applications (if a P4 SB is chosen)
    • Hardware switch related issues
    • Virtual switch related issues (local and e2e tests), in the worst case it might not be a full blocker, but would slow development-release cycle down significantly
    • What can we also do to try to mitigate these risks?
      • Before bulding the production grade code, we should try to explore in depth these parts.
      • Network engineering team should specify the desired pipeline and the earliest it can be tried out and constanly explored the better

Example with aiop4 and p4_l2ls (L2 learning switch)

• First proof of concept milestone showing a NApp using P4Runtime API on Kytos-ng; asyncio-oriented
• Simple (old) throw-away prototype p4_l2ls NApp with aiop4 (with p4runtime v1.4.1) and l2ls running on Kytos-ng 2024.1.2
• mininet linear,2 topology except with simple_switch_grpc switches insted of OvS switches

*** Starting 2 switches
s1 Starting P4 switch s1
simple_switch_grpc -i 1@s1-eth1 -i 2@s1-eth2 --no-p4 --log-console --device-id 1 --thrift-port 9090 -- --grpc-server-addr 0.0.0.0:9559
switch has been started
s2 Starting P4 switch s2
simple_switch_grpc -i 1@s2-eth1 -i 2@s2-eth2 --no-p4 --log-console --device-id 2 --thrift-port 9091 -- --grpc-server-addr 0.0.0.0:9560
switch has been started
**********
h1
default interface: eth0 10.0.0.1        04:00:00:00:00:01
**********
**********
h2
default interface: eth0 10.0.0.2        04:00:00:00:00:02
**********
Ready !
*** Starting CLI:
mininet> 
mininet> h1 ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=7.44 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=2.53 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=2.93 ms

2024-12-20 16:21:49,892 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Setting up config for up4rt:9559:1
2024-12-20 16:21:49,892 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Setting up config for up4rt:9560:2
2024-12-20 16:21:49,920 - INFO [aiop4.client] (MainThread) Starting stream_control for up4rt:9559:1
2024-12-20 16:21:49,921 - INFO [aiop4.client] (MainThread) Starting stream_control for up4rt:9560:2

     _   __      _
    | | / /     | |
    | |/ / _   _| |_ ___  ___          _ __   __ _
    |    \| | | | __/ _ \/ __| ______ | '_ \ / _` |
    | |\  \ |_| | || (_) \__ \|______|| | | | (_| |
    \_| \_/\__, |\__\___/|___/        |_| |_|\__, |
            __/ |                             __/ |
           |___/                             |___/
    
    Welcome to Kytos SDN Platform!
    Kytos Version.: 2024.1.2

kytos $> 
kytos $> 2024-12-20 16:22:20,158 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Client up4rt:9559:1 learning b'\x04\x00\x00\x00\x00\x01' in b'\x01'
2024-12-20 16:22:20,159 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Client up4rt:9560:2 learning b'\x04\x00\x00\x00\x00\x01' in b'\x02'
2024-12-20 16:22:20,160 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Client up4rt:9560:2 learning b'\x04\x00\x00\x00\x00\x02' in b'\x01'
2024-12-20 16:22:20,162 - INFO [kytos.napps.kytos/p4_l2ls] (MainThread) Client up4rt:9559:1 learning b'\x04\x00\x00\x00\x00\x02' in b'\x02'

• Checking flow entries installed on IngressImpl.dmac table on the first switch:

simple_switch_CLI 
Obtaining JSON from switch...
Done
Control utility for runtime P4 table manipulation
RuntimeCmd: table_dump IngressImpl.dmac
==========
TABLE ENTRIES
**********
Dumping entry 0x0
Match key:
* ethernet.dstAddr    : EXACT     040000000001
Action entry: IngressImpl.fwd - 01
**********
Dumping entry 0x1
Match key:
* ethernet.dstAddr    : EXACT     040000000002
Action entry: IngressImpl.fwd - 02
==========
Dumping default entry
Action entry: IngressImpl.broadcast - ab
==========

Quick comparison between BFRuntime API and P4Runtime API

Although they both are protobuf specs, they're not interchangeable, there are some common parts, but there are more messages differences than similarities:

Feature	p4runtime.proto	bfruntime.proto
Target Hardware	Vendor-agnostic (std)	Barefoot Tofino switches
Features	Standard P4 features	Tofino-specific features and optimizations

Additional major TNA features:

• Advanced parsing and pipeline configurations
• Port status events via gRPC stream
• Pipeline insights compilation tooling
• Hardware specific counters
• Ports aggregation and load balancing
• Custom additional fixed functions that can be used
• Mirroring and telemetry

Port status on P4Runtime API would need another API like gNMI or via thrift and polling

Next Steps

• This document went over the main design considerations to support new remote southbound APIs on Kytos-ng
• Decide on which SB to integrate next when AmLight use cases are defined and confirmed with certainty, and plan the scope and break down the new epics
• Is Kytos-ng current architecture and its incremental evolution suited for what's desired while leveraging its strengths? or should we start reconsidering other projects?

[viniarck]

Random rant: I really wish P4 Runtime API still succeeds in becoming a standard API widely adopted API for P4 devices.

Imagine if P4 Runtime API were to get the best parts where BF Runtime API pioneered and got right, now that its p4c compiler backend modules and SDE are open source, maybe that can happen one day, although of course BF Runtime API still has many Tofino Native Architecture concepts where it might too opinionated for a std (or not), we'll see how it'll unfold.

[viniarck]

TODO: add new table to mapping which OpenFlow event messages are being used, just so it can be used to check if another SB provides something somewhat equivalent