My customer uses Sonatype Nexus as their artifact repository for all kinds of packages and also for Docker Containers. Since there are a few Microsoft .Net teams are moving towards Docker, the need for Docker containers arose as well.
So we created a Windows Base container and tried to push this to the Nexus repository. But unfortunately, this threw an error.
58c642e95314: Skipped foreign layer 6c357baed9f5: Skipped foreign layer errors: blob unknown: blob unknown to registry blob unknown: blob unknown to registry
In first instance I thought that it was that Sonatype did not support Windows Containers or that the installation was strange, so I started fiddling with it. (Spoiler alert: it is not a Windows Container problem, it is a Foreign Layer Problem.)
To get started easily I spun up a Linux Machine on Azure with Docker Installed. On this machine I ran the Nexus3 Docker Container, to get a fresh Nexus3 installation
docker run -p 8081:8081 -p 8082:8082 -p 8083:8083 sonatype/nexus3
After that I configured my Nexus as described in this post. I ended up with a private repository on port 8083 and a proxy for Docker Hub on port 8082 on my public IP address of my Linux machine on Azure.
To make this accessible, the ports on the network security group should be opened as well. So in the Azure Portal (or with the CLI) open the inbound ports for 8082, 8081 and 8083 to access your Nexus server
When everything was set up correctly, I set up my Docker for Windows Client to use Windows Containers, and logged in to the registry.
docker login -u admin -p admin123
linuxmachine.westeurope.cloudapp.azure.com:8083
(note that the default Nexus Username / Password is admin/admin123)
In order to push an images to this registry you have to tag a default image to this registry
docker tag microsoft/nanoserver
linuxmachine.westeurope.cloudapp.azure.com:8083/nanoserver
Then push the image
docker push linuxmachine.westeurope.cloudapp.azure.com:8083/nanoserver
Now you get either the error I showed you above or
http: server gave HTTP response to HTTPS client
So the error consists of 3 parts that are interrelated. The first part is about the HTTPS error. You need to specify to your Docker client that you want to use an insecure registry. You can do that by either adding this to the Daemon Tab in the Docker Settings on your client, or by adding it to the daemon.json file by hand located in C:\ProgramData\Docker\config
Add your registry to the “insecure-registries” array in this file
{
"registry-mirrors": [],
"insecure-registries": [
"machine.westeurope.cloudapp.azure.com:5000",
"machine.westeurope.cloudapp.azure.com:8082",
"machine.westeurope.cloudapp.azure.com:8083"
],
"debug": true,
"experimental": true
}
When you add this and restart Docker (Docker/Settings/Reset/Restart) you can push and get the “blob unknown” error. You also see the “skipped Foreign Layer” pooping up. Those 2 are related. Foreign Layers are actually links to layers on another URL. In case of Windows Images, the base layers are not distributed (licensing). And now comes the good part, Nexus does not support Foreign Layers, as described in this issue.
So it is not a Windows Container problem, it is a Foreign Layer Problem. Now how to solve it. The same as with the insecure registries, you can force your client to push the foreign layers in any case. You can do that by adding the following lines to your daemon.config
"allow-nondistributable-artifacts": [ "rvolinuxdocker.westeurope.cloudapp.azure.com:5000", "rvolinuxdocker.westeurope.cloudapp.azure.com:8082", "rvolinuxdocker.westeurope.cloudapp.azure.com:8083" ]
This forces the client to push the image. After restarting the images pushes fine ! and can be pulled as well. For more information on the Docker Daemon, look at this documentation
Hope this helps!
I work as Lead Consultant at Xpirit. My personal motto is: "Continuous improvement!" I seek challenge in the fact that there is always a better way. I always try to push that extra step to achieve optimal results. Currently, Application Lifecycle Management (ALM) is the perfect way to implement my motto. By using the right methods and tools I help organizations, departments, processes and people to improve in the field of software development.
Because of my contributions to the community I was rewarded with the Microsoft Most Valuable Professional (MVP) award in 2012, 2013, 2014 and 2015