Hosting firmware builds on GitHub? (fetch/CORS issues)

[T-vK]

I am using GitHub Actions in order to automate my build process. So basically every time I push a change, it automatically creates a new build and adds a release to my GitHub repo making the binaries accessible. That in itself works great and the logical next step would be to utilize GitHub Pages in order to flash any release. But that is where I'm running into a big issue: I cannot fetch an asset from a GitHub release from the JavaScript of a GitHub Pages site because of how GitHub has set up their CORS rules.

When you search for that CORS limitation you actually quickly find other users of esp-web-tools running into the same issue.

I would like to know if anyone has found a way to make this work in some way.

The only way I could think of would be to store the builds directly in the git repository, but that would unnecessarily bloat the repo and is bad practice to begin with. Unfortunately, the only other way I see is to host on some paid storage solution, which is less than ideal for an open source project that doesn't make any money.

[T-vK]

I think I have found a ... somewhat acceptable workaround.
The two main issues with storing binaries directly in a git repository are:

1. Performance (slow cloning speeds etc)
2. Bloating the history

In order to work around that you can use a separate branch and essentially disable the commit history on it.

I'm using the gh-pages branch to store all my firmware builds and whenever I push a new firmware into it, I write it into the first and only commit that exists on that branch.
This can be achieved quite easily like this:

git checkout gh-pages
git add ./firmware/
git commit --amend --no-edit
git push --force

In order to still have proper source control on the actual html code of the GitHub Pages site, I store my index.html on the main branch and copy it into the gh-pages branch on every push.

I am of course not doing any of this manually. I used GitHub Actions to automate it all.

I don't know if it makes sense to link my project here because it will undergo a lot of changes the coming weeks, but here it is nevertheless: https://github.com/T-vK/ShowboxInterceptor

I also managed to dynamically create manifests on the fly using unmodified esp-web-tools. The trick is to intercept the fetch request and inject a generated response. See index.html in my repo.

If anyone has a better idea on how to do all this, I'm very open to having a discussion about it.

[tobiasfaust]

Hi,
an alternative is to setup a little cors-anywhere web service. With this, you are able to fetch files from anywhere without cors issue. A demo can be found here: https://github.com/Rob--W/cors-anywhere

Cors-anywhere exists as docker too ;)

[T-vK]

Yes, that is of course true, but you will have to rent and maintain a server/service in order to do that.
Keeping everything on Github eliminates all the maintainance work and ensures scalability, availability, stability and performance without you having to do anything. It will simply keep working even after you die or abandon your project.

[tobiasfaust]

ok, you are right.
But on the other hand it works perfectly with github pages without a CORS issue if you store the binary files in your Repo
I´m using the webinstaller in my repo. Binaryfiles will be stored in a sparate web-installer branch which is defined for Github pages

-> https://tobiasfaust.github.io/SolaxModbusGateway/

[T-vK]

Yes, if you store the binaries in the branch used for github pages, then it just works and no additional cors proxy is required.
That's how I'm doing it in my project too: (Branch, Page) as mentioned in this comment.

Even though it is generally considered bad practise to commit binaries, I highly recommend doing it this way considering there is no good alternative.
One more thing to avoid potential issues where a git clone might take forever because of all the committed binaries would be to store the binaries and github pages for the project in a separate github repo.
But it does of course come with the downside that it is potentially confusing to other people trying to understand it.

[Jason2866]

Been there storing the release binaries in an separate branch. Worked. After a while got a friendly note from github team (not a bot, real person) which explained git is not for storing binaries and the repo size has become to big. We got a long enough time line to change this.

So everyone considering storing binaries releases in a branch keep in mind github may notice this and asks you to stop that.

[T-vK]

Thanks for the heads up, can you give us information on how big the repository was? I think it could be a viable option to just keep the last 10 releases in that branch and tell users that if they need to flash an older version, they have to download it manually.

[Jason2866]

Well it had about 10GB ;-) Unofficial allowed are 5 GB. Since we become in focus, we had no choice to do it "clean" We are storing now only the latest build version in the repo.
Older ones are available via release section. User can load via the WebInstaller too.
Needs to install an CORS Unblock add in for the bowser.

[T-vK]

Thanks, that's good to know. Just out of curiosity, how did you manage to get to 10GB? I mean if one binary is around 1MB (that's the size of my ESP32-S3 builds), that would be like 10000 binaries, right?

[Jason2866]

Well Tasmota offers some different build variants plus the factory image. We stored all built firmwares from every commit. Reaching 10GB isn't that hard :-)

[T-vK]

Okay, that makes sense. Thank you :)