Interview – NXP Linux BSP and Timesys Vigiles Maintenance Service & Security Updates

I’ve been interviewing Ed White, Manager of NXP’s Professional Support and Engineering Services, and Akshay Bhat, Director of Engineering, Security Solutions at Timesys by email to find out more about NXP Linux BSP development process, and how Timesys can help to keep it updated and secure with its Vigiles service.

Q1. CNX Software readers recently discussed NXP Linux BSP update status. One person specifically noted Linux 4.14.98 used in the BSP was well over a year old, and there were various opinions about the topic, including one person suggesting NXP only provides a stable BSP and it was the ultimate responsibility of the customer to merge Linux security patchsets. Could you explain the typical development process for NXP Linux BSP, and why the company chose not to update the patchsets regularly?

Answer: The kernel strategy for NXP’s i.MX family BSPs closely follows the annual cadence of kernel.org’s LTS kernel selection. As soon as kernel.org establishes the next official LTS kernel version, NXP transitions our internal development to that particular kernel. However, the migration of the kernel is only one aspect of our next major release. There may be a number of associated updates to be included, such as a new version to Yocto, updates to U-Boot, and many other package changes we integrate into the Yocto release specific to the i.MX BSP. These factors, plus our rigorous testing process create an inevitable delay between the community version of the latest LTS kernel release and NXP’s i.MX board support package (BSP) based on that same kernel.

We must also consider a number of other factors that come into play between our planned cadence of Linux LTS kernel updates. NXP may introduce new products, or there may be updates to various packages, and of course, there are issue resolutions (including LTS minor version updates) to be considered. Our engineering team must balance all these factors while maintaining consistent quality standards for the entire i.MX product family being supported by each BSP release.

We (NXP) provide support for each new LTS kernel for at least one (1) year after delivering the initial general availability (GA) release of that version. From the change factors I just described, one can expect to see several minor updates in between the major LTS kernel version updates. These typically come as complete BSP releases, and occasionally as a specific patch update. Customers developing i.MX-based products should monitor NXP.com to ensure they have the latest BSP release and updates.

I read a discussion from one of your readers about an NXP partner product released in May of 2020 based on our L4.14.98 BSP. The discussion implied NXP has not provided a more current Linux kernel version BSP for i.MX – which isn’t true. Since releasing the L4.14 kernel version BSP, We have delivered an L4.19 LTS kernel version BSP as well as a few minor version updates in-between. In late 2019 the development team was already migrating to LTS kernel version L5.4, which was initially released by us in Q1, 2020.

In defense of any NXP partners or customers developing products based on i.MX products, they each have their own development and test cycles and typically select the NXP BSP version that matches their production schedule and goals. Despite having a product “officially” developed on a specific NXP BSP kernel version does not mean they haven’t been monitoring their BSP for the latest vulnerabilities and applying the appropriate updates. Let’s discuss how NXP is enabling the capability to have the latest and most secure SW platform – without a wholesale BSP update – by using the Vigiles tool powered by Timesys.

Q2. Soon after this discussion, I discovered Timesys Vigiles service that makes sure the Linux BSP is secure with the customer’s own patches. Could you provide an overview of the development workflow?

Answer: NXP/Timesys has two offerings to help maintain device security:

  1. Vigiles is a security & vulnerability notification and reporting tool for monitoring your software. Vigiles includes many features like links to patches, mitigation information, alerts for new CVEs, collaboration tools, etc. Using information provided by Vigiles end customers can secure the BSPs themselves (e.g.: patch/upgrade packages and kernel).
    Timesys Vigiles NXP Linux-BSP Security Updates
    Click to Enlarge

    Note: Vigiles is pre-integrated in NXP i.MX Yocto BSP releases. Customers can register for a Vigiles Prime 30-day evaluation to experience all the features of the tool in their own project. (After 30 days, it converts to a less feature-rich, free version.) See sample demo report.

  2. The BSP Maintenance service is a managed service whereby customers provide their hardware and BSP sources along with any custom patches to NXP/Timesys team for security maintenance. As part of BSP maintenance service customers get:
  • A subscription to Vigiles Prime: Security & vulnerability notification and reporting tool for monitoring your software. Vigiles Prime is used to collaborate between customer and BSP Maintenance teams on planned security releases.
  • Complete BSP update (software release) twice a year (by default, and the cadence can be changed if needed) on a mutually agreed timeline
  • Minor kernel version upgrade for security and bug fixes
  • User space security patching & package updates
  • Each update is validated and tested on the customer’s hardware
  • Release notes and test reports included with each update
  • Customer-provided hardware is maintained in our Timesys board farm
  • In the event something critical happens between updates…
  • On-demand update for emergency security fixes (one per year included)
NXP Linux BSP Maintenance Workflow
BSP Maintenance Workflow – Source: Lifecycle Maintenance of Your BSP presentation slides

More about the process answered under Q4, BSP maintenance section.

Q3. New CVE (Common Vulnerabilities and Exposures) get discovered daily, so the system needs to be updated regularly to be kept secure. How often does a product managed through Vigiles typically get security updates?

Answer:

  • Vigiles subscribers: Vigiles’ curated database is updated on a daily basis. Customers can subscribe to daily/weekly/monthly notifications to be alerted about CVEs applicable to components in their BSP. Customers will need to use the information available in Vigiles to DIY patch/upgrade components to address the vulnerabilities. Since customers are driving the process, they also choose the exact time and frequency to roll out security updates in their products.
  • BSP maintenance subscribers: Customers subscribing to the full BSP maintenance offering have the option to select their desired upgrade cadence. The typical options are: monthly, quarterly, semi-annual, or yearly. The cost of BSP maintenance varies based on the selected option. The service also includes one emergency release for addressing any high severity vulnerability outside of the scheduled release cadence.

Q4. Some open-source tools, including the Yocto Project, come with CVE checkers. That means some customers could apply the latest Linux patches for Linux 4.14.xxx and run a CVE checker to find out potential vulnerabilities themselves. Would Vigiles still make sense in that case? What would be the added benefits of the offer?

Answer: Yes, the Vigiles and BSP maintenance services still make sense for customers like you are describing.

Our Vigiles services provide significantly more accurate and timely vulnerability data and more accurate software component analysis than open-source CVE checkers, which dramatically cuts analysis and mitigation time. Our services also include the collaboration, mitigation, and update tools and services that are needed for an efficient security and maintenance process. These are based in large part on industry best practices derived from thousands of software projects we have supported over the years, all at a very low-cost.

While Vigiles offers the above benefits over open-source CVE checkers to reduce the burden in accurately monitoring and finding fixes, there is still effort/skill required for patching, resolving conflicts, and testing. Hence, customers might want to go with the managed BSP maintenance service to completely offload the BSP security maintenance burden to NXP/Timesys. This would provide significant cost savings and result in a more secure BSP.

Someone taking a DIY approach with open-source CVE checkers can certainly try to recreate that and reinvent the wheel, but it would not make sense to do so.

Linux BSP Maintenance Costs DIY vs Vigiles
Estimation of BSP Maintenance Costs per Board

To provide specifics about these added benefits:

  • Vigiles:
    • Vulnerability Data Accuracy: The open-source tools rely on the NVD database. The NVD data has quite a few false positives and missed CVEs due to data entry errors and/or outdated information. E.g.: When a CVE fix is backported to an LTS kernel version, the NVD database is not updated with this information. (Slide 17, 22 of ELC presentation).
      CVE False Positives
      Vigiles pulls vulnerability data from multiple sources such as NVD and Security Bulletins (Ubuntu, Debian), which is run through in-house-developed curation algorithms to reduce false positives (by cross-verifying available fixes against git commits to confirm affected version ranges, etc.), map vulnerabilities to package config options and limit to affected platforms. In addition, the Timesys security research team manually monitors vulnerabilities in packages from build systems such as Yocto and Buildroot, SoC vendor advisories, and mailing lists such as oss-security, and adds/curates vulnerability information as needed. All of these steps lead to improved accuracy and coverage while reducing false positives in our curated database (up to 40% improvement over NVD for certain packages).
    • Vulnerability Data reporting delay: Oftentimes, there is a delay in when a vulnerability is reported to when it gets published and analyzed by NIST NVD. This can result in high severity vulnerabilities being reported in the news but not in open-source tools relying on NVD (Slide 23 of ELC presentation). Since Vigiles augments multiple sources for vulnerability data, we report up to four weeks earlier than NVD in some cases.
    • Tool accuracy: The CVE checking tools themselves have issues with the way NVD feeds are parsed and versions are compared. Recent improvements have been made but have not been backported to older versions of Yocto, leading to a false sense of security. (Slide 19, 20 of ELC presentation).Vigiles is maintained for older releases of Yocto and Buildroot and does not suffer from similar deficiencies.
    • Ease of use: Open-source tools such as the one built into Yocto do not give a high-level overview of which package has which vulnerability. Further, they don’t differentiate build time vs. run time packages, leading to a lot of manual and cumbersome investigative clean-up effort. Vigiles provides an easy to read a summary by package along with a detailed section that can be searched/sorted. Further, we only report CVEs for packages installed on the target.
    • Filtering: Most times, even though a vulnerability is applicable to a package, the feature resulting in the vulnerability might not be enabled in the build. Vigiles lets you upload kernel and u-boot configs and automatically filters out CVEs not applicable based on config options being used reducing analysis time by 4X. Further, one can filter based on severity or attack vector to help prioritize the ones that need to be addressed first.
    • Links to patches: Finding CVEs is only a part of the process. The real benefits come from identifying if a fix is available, whether a newer version has been released with the fix, has it been backported, and so on. This detailed analysis is a time-consuming process when you are dealing with hundreds of CVEs.Vigiles also gives links to patches and suggests which version of the software to upgrade to address the vulnerabilities.
    • Value-added features: Vigiles offers team collaboration and communication tools such as adding notes, whitelisting CVEs as not applicable to a project, and so on. It also highlights changes from one software component scan to another, gives a history plot, can send email alerts on the desired frequency, export reports in various formats, search the curated database, to list a few of the many features. All these features were added based on customer feedback, our own need to be efficient when addressing vulnerabilities and insights gained from the thousands of software projects Timesys has supported over the years.

Vigiles is more than a monitoring tool, it provides an end-to-end workflow to address vulnerabilities at volume. This enables a device maker to increase product security at a significantly lower cost than attempting to DIY with open source CVE checkers.

  • BSP maintenance: The challenges/effort associated with DIY security maintenance is as follows:
    • Kernel security: The upstream kernel maintainers do a great job of backporting security and bug fixes to LTS branches. So migrating to the latest minor version of the LTS kernel typically addresses most security vulnerabilities. Taking 4.14.98 i.MX release as an example, there are ~6659 NXP patches on top of the mainline 4.14.98. Under BSP maintenance we would rebase the NXP patches on top of the latest 4.14 LTS kernel (eg: 4.14.183). This typically results in many conflicts since the same code can be modified by both NXP patches and in 4.14.183. We analyze which changes to keep from upstream and which ones to keep from NXP and resolve the conflicts.DIY route of security maintenance: As you can see, resolving conflicts requires a deep understanding of how the driver works. This is a highly specialized skill-set that is not widespread among end-customers who mainly consume BSPs.
    • User space security: Many times, upgrading a package might result in API changes that require end-user application changes. We work with the customer and backport or upgrade packages on a case-by-case basis, ensuring minimum burden to the end customer.DIY route of security maintenance: If patches are available, investigate if they can be applied cleanly. If they cannot, you then need to figure out how to backport and ensure a given patch addresses a vulnerability without breaking any functionality.
    • Testing and Reports: A driver test framework along with driver tests is made available to end customers and is pre-configured to test the peripherals based on end-customer hardware. NXP/Timesys runs this test suite with the updated BSP on the customer hardware. For user space, we run tests from either Yocto ptest or built-in test suites available in packages and verify the tests continue to pass. We provide vulnerability and test reports as part of the release which can be used for any required compliance (e.g.: FDA).DIY route of security updates: Customers need to write or integrate their own driver tests and configure them for their platform and validate the BSP. Further, generating testing and before/after vulnerability reports is a cumbersome process without prior investment in automation.
    • Support: We work as an extension to the customer team and are available for answering any security vulnerability related questions.DIY route of security maintenance: Engage with the open-source community and hope for a response or hunt for information on mailing lists, etc.

Having the right tools, automation and expertise is key to bringing the cost of BSP maintenance down and reducing security risk. In short, NXP/Timesys services capture industry best practices to help bring more secure products with reduced maintenance cost allowing end customers to focus on product value add.

Q5. Vigiles is specifically targeted at NXP Linux BSP, but what about other SoCs with mainline Linux support or a BSP with a Linux LTS kernel? Would you be able to provide support? If so, would the silicon vendor need to be involved, or any client could hire your services for support?

Answer: For NXP BSPs, Vigiles and/or BSP maintenance can be purchased directly through NXP.

For other vendors’ BSPs, Vigiles and/or BSP maintenance can be purchased through Timesys. The silicon vendor does not need to be involved, although Timesys has longstanding relationships with many vendors. Any end-customer can hire Timesys for their specific security or BSP maintenance needs. Timesys has expertise in maintaining BSPs on various SoCs. Further, if an end-customer is not on an LTS kernel, Timesys can first migrate them to an LTS version under an additional service contract and then start BSP maintenance to keep the device secure.

Biography of interviewees

Ed White NXP Ed White, Manager of NXP’s Professional Support and Engineering Services. Ed has worked in the semiconductor business for over 24 years in predominantly marketing and product management roles. An employee of NXP since 2011, he is currently managing NXP’s global Professional Support and engineering services business, based in Austin, TX. Ed has an undergraduate degree in Computer Science (BSCS) and a Master’s degree in Business Administration (MBA).
Akshay Bhat Timesys Akshay Bhat, Director of Engineering, Security Solutions at Timesys. Akshay has experience with embedded systems that spans a broad range of industries with a focus on board bring-up, driver development, and software security. Akshay received his MS in Electrical Engineering from NYU Polytechnic University.
Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK 5 ITX RK3588 mini-ITX motherboard

13 Replies to “Interview – NXP Linux BSP and Timesys Vigiles Maintenance Service & Security Updates”

  1. Not sure to understand very well, in addition to buy an expensive SoC you should also subscribe to keep it secure?
    I mean, If I choose an NXP SoC it’s in order to have support and software update.

    1. Once you buy a chip from NXP and integrate it into your custom design, who is going to keep your custom code patched and up to date? The actual security patches are free for everyone to use.
       
      But who does the necessary labor to apply these patches into your custom software environment? If you have the skills, then you can do this yourself. It does take a significant amount of time to track these patches and apply them onto your custom code. With this service you are hiring someone to do that work for you. The ‘subscription’ is for the labor needed to do the work necessary to apply these free patches into your custom code.
       
      Check out the pricing — they price by the number of boards they are working on. Those boards are your custom hardware/software environments. This service is keeping your custom code up to date with security patches.
       
      NXP also provides periodic updates to their generic BSP that integrates the security patches. You can also use that method where you take each new BSP release and redo all of your custom changes to it.
       
      Embedded systems are not like the desktop where you can apt-get the updates. With embedded systems you have to construct device specific OTA update images and then distribute them (like your phone uses). Those OTA update images are specific to each phone design. So who is doing the labor to make these device specific OTA update images? You can hire timesys to make them for you.
       
      A common problem in the embedded world is that the company that manufactures the device never bothers to update it with security patches. Companies always have a thousand excuses for why they don’t do that. Now there is a relatively inexpensive option which allows the company to offload that work onto timesys.
       
       
       

      1. Cost is about same for one board as one engineer, but ideally you would need someone to do the peer review. Someone to discuss issues and find solutions, etc. So if you are having 1 to 3 or 5 boards this service makes a lot of sense, unless you are having a capable department already that can take over this work as well, or maybe just needs one full time position more to cope with it.

      2. NXP Provides periodic updates but it’s not tagged as “GA” release, so I’m not sure if it’s fine to switch to these releases.
         
        Also there BSP update bump from 4.14.98, to 4.19.x or recently to 5.4.x. Which is a bit more complex to update.
         
        I can finely maintain my custom patches but I certainly can’t take the ~6659 NXP patches. I would expect NXP to release updated version of the 4.14.x branch with there kernel protected again latest CVE without paying extra fees.
         

        1. No vendor is going to provide a BSP continuously updated to the latest CVE. That’s because CVEs come out everyday and these BSPs get tested. Not even Google does a new release for each CVE.
           
          The vendors will release periodic updates, that’s the best you can ask for and NXP is doing that. You then need to take those period vendor updates, apply the CVEs since then, reintegrate your custom changes, test, build OTA image, push OTA image to devices. You can do that cycle as often as you want. That’s the service timesys is offering, they’ll run that cycle for you. Google does this cycle once a month for Android.
           
          Any cycle faster than that requires a system like Ubuntu Livepatch where they patch running systems without pushing an updated image.
           
          There is a trade off here. If you run the cycle faster the customers are not going to like their device rebooting everyday. Plus OTA updates can fail. Imagine the problems if you accidentally push an OTA that won’t boot. So for very quick patching you need to switch over to the Livepatch type scheme.
           
          Maybe NXP could do periodic releases of their older BSPs with all of the patches available at the time of that release applied. Not sure how much effort that is versus how many customers would use. Instead I think they’d tell you to track the kernel bumps from 4.14 to 4.19 to 5.4….
           
          Another solution here is to simply not use the BSP. NXP keeps mainline up to date. You can work off from it instead of the BSP.
           
          NXP is doing a great job in making security updates available. Most other vendors, especially Chinese ones, are far, far worse. Consider Allwinner a master practitioner of “port and forget”. They do one BSP release and then never update it. Plus they don’t put their code into mainline. Allwinner could easily implement better security practices, but they simply don’t care about what happens after you pay for the chip.

          1. Yes, I agree with you and I’m fine with the actual periodic updates NXP make and the stuff they do mainline but they are not flag as “GA” release => SoM vendor doesn’t want to move on it => If you move on it you don’t have any support.
             
            I hope NXP will keep this strategy and not change their mind saying “if you want more updates use Timesys Vigiles”.
             

  2. So if I pay Timesys to protect and maintain my product’s OS, and they fail to do so, how am I covered against monetary and reputation losses? Is there an independent insurance policy or bond? If not, I am going to have to spend a lot of time and money watching Timesys to make sure they are doing what I could be doing in the first place! And don’t forget, one way or another I will end up paying the premiums for any bonds or insurance policies. Those premiums will skyrocket if Timesys screws up, not only in the work they do for me, but in the work they do for ANY of their customers. That’s a lot of risk to carry, and risk has value. Finally there is the legal costs of having to go after Timesys for restitution if there isn’t some sort of protection in place.
     
    Liability is the 10,000 pound elephant sitting in the room here, and nobody seems to be addressing it. Taking a look at the prices and considering the potential liability cost of using Timesys, I thing I would prefer to keep the software maintenance in-house and self insure.

    1. Of course you are going to do that if you are a multi-billion dollar company. But there are a lot of smaller companies without resources at that level. This is aimed at those thousands of companies that ship product and then never patch it.

  3. Thanks for this interview Jean-Luc, but this is completely retarded and just confirmed to me never to use one of these BSPs since they have zero understanding of how the kernel works. Better wait for their devices to be fully supported in mainline (and yes, at least they’re doing a good job at this and they should be thanked for this).
     
    First, the first guy clearly shows he doesn’t understand what maintenance means, he doesn’t even understand your question. He only speaks about major upgrades and not bug fix updates. He even speaks about QA on a single upgrade, then his job is done! <b>This</b> is exactly what must be stopped, it’s how operating systems used to work in the 90s when they were not connected to the net and updates were shipped on floppies, but things have changed over the last 25 years and users want to apply fixes to their products, and don’t want to be forced to perform major upgrades in field. The new LTS kernel cycles just reflects this.
     
    Second, they’re speaking about backporting fixes for CVE. <b>This</b> is exactly what must not be done! CVEs are a marketing circus which only serves two purposes nowadays:

    • inflate one student’s ego to make him the “owner” of a CVE
    • tag desired fixes to bypass stupid distro backporting rules

     
    More background here: https://lwn.net/Articles/801157/
     
    What is needed is to take <b>all</b> patches from the LTS tree, because they were validated by their authors as a whole. Some explicitly warn that “patch X or Y is needed with it”, and those cherry-picking “CVEs” will miss these ones which are apparently unrelated and end up with a totally broken kernel. Running CVE reproducers to test a kernel is ridiculous, as it doesn’t prove that nothing else broke.
     
    For having done many kernel backports myself, being absolutely certain they were OK and learned during reviews by the patches authors how many times I was wrong, I will <b>never ever</b> trust a set of backported fixes that were not reviewed in context by their authors anymore.

    1. You are absolutely right, but still its worlds ahead of what we usually are getting now. And probably much better than what the average it department is capable of. Seeing what an accumulated experience and knowledge we are hoarding at my current work place, even though we are not in IT, but it’s similar, trust me.

    2. Hi Willy, I am Akshay from Timesys (from the interview).
       
      Regarding the BSP Maintenance offering and the context of the Linux kernel: The process we follow is to rebase NXP patches and customer patches on top of the corresponding latest mainline LTS kernel. We do not backport or cherry-pick CVE fixes for the same reasons suggested by the kernel maintainers in the lwn article you referenced. In-fact I had presented a similar talk at the very same ELC conference in which Greg KH (LTS kernel maintainer) presented his analysis (links are in the above article).
       
      The BSP maintenance process is illustrated in the BSP Maintenance Workflow (refer to the kernel box in https://www.cnx-software.com/wp-content/uploads/2020/07/NXP-Linux-BSP-Maintenance-Workflow.jpg)
       
      Taking a specific example we further explain the same concept in Q4 under BSP Maintenance:
      Kernel security: The upstream kernel maintainers do a great job of backporting security and bug fixes to LTS branches. So migrating to the latest minor version of the LTS kernel typically addresses most security vulnerabilities. Taking 4.14.98 i.MX release as an example, there are ~6659 NXP patches on top of the mainline 4.14.98. Under BSP maintenance we would rebase the NXP patches on top of the latest 4.14 LTS kernel (eg: 4.14.183).
      So all of the patches in the LTS tree are being included as whole.
       
      Hope this helps clarify the offering 🙂
      Thanks,
      Akshay

      1. Thank you for clarifying Akshay! So OK from your description above it looks perfectly fine (and rare enough in the industry), so good job!
         
        However I must let you know that this process is not quite clear from the initial responses which instead make one think the focus is essentially on new major LTS versions. But maybe that’s the difference between the speech that managers want to hear and the one developers want to hear. And similarly the diagram makes one really think you’re backporting CVE fixes (which I’m sure some customers believe that’s what they need and probably they want to see that written somewhere).

Leave a Reply

Your email address will not be published. Required fields are marked *

Boardcon Rockchip and Allwinner SoM and SBC products
Boardcon Rockchip and Allwinner SoM and SBC products