Running OpenGL Shaders on the Raspberry Pi - Posts

So I've accepted the mission to run shaders from Shadertoy, on the Raspberry Pi! The Linux Graphics Stack. I use a Raspberry Pi 4 as my ... PostsTagsAboutRSS AshaderfromShadertoyrunningfullscreenontheRapsberryPiofficialtouchscreen⊕TL;DR:Ifyoujustwanttotryit,youcanheadtotheprojectrepository.AsfarasIcanremember,I’vealwaysfeltattractedbycomputergraphics. IguessIapproachthefieldfromtheangleofmyMathematicsbackground,asIseeitasauniversallanguagebetweenmachines,arts,andpossibly,nature“Howcanitbethatmathematics,beingafterallaproductofhumanthoughtwhichisindependentofexperience,issoadmirablyappropriatetotheobjectsofreality?”—AlbertEinstein..Asaprojectionofthatlanguage,IfindtheOpenGLShadingLanguageaninterestingcase,attheintersectionofcomputingandvisualarts,that’sconciseandexpressiveenoughtodemonstratehumaningenuityandcreativity. Youcanfindmebrowsingforhoursonshadertoy.comShadertoyhasbeencreatedbyIñigoQuilez,thatpublishesveryinterestingarticles,tutorials,andotherawesomeresourcesatiquilezles.org.,mesmerizedbythevisuals.SinceIbuiltmyKubernetesclusterwithRapsberryPiayearago,I’vekeptsomewhereinacornerofmyhead,theideaIcouldusethetouchscreenmonitortoplaywithOpenGL,makingthecoolestclusterever,tobeginwith😎,andturningiteventually,intoasmallgamingdevice🕹👾Ascrazyasitmaysounds,it’spossibletodevelopgamesinGLSL(clickontheimagestotryit): .SoI’veacceptedthemissiontorunshadersfromShadertoy,ontheRaspberryPi!TheLinuxGraphicsStackIuseaRaspberryPi4asmycluster’smainnode,that’sconnectedtothetouchscreenmonitor. ItrunstheLiteversionofRaspberryPiOS,whichmeansthereisnowindowingsystemavailable,likeX11.IalsohavesomeRapsberryPi3,soIwantedtohaveasolutionthatwouldworkontheseaswell,andpossiblyonanyotherLinuxdevicewithGPUhardware.ThefollowingdiagramgivesagoodunderstandingoftheLinuxgraphicsstack:Mesa:APIs+DRI/Gallium3Ddriver(libGL)libGL-mesa-swx11libGL-mesa-glxlibOpenVG-mesalibGLES-mesalibEGL-mesa(Wayland)libEGL-mesa-driverslibGBM(Modules)libGL-mesa-DRI(X.Org)X-serverX.OrgServerdisplaydriverxserver-xorg-video-nouveauxserver-xorg-video-nvidiaxserver-xorg-video-radeonDRMLinuxkernellibDRM-intellibDRM-radeonlibDRM-nouveaulibDRM-freedrenohardwarespecificUserspaceinterfacetohardwarespecificdirectrenderingmanagerCPU&registers&L1&L2&L3&L4&mainmemoryGPU&registers&L1&L2(&graphicmemory)ApplicationsToolkitslibDRM3D-gameengineDDX-driverlibX/libXCBProprietaryOpenGL4.2driverlibGL-nvidia-glxlibGL-fglrx-glx"libGL"RenderingAPIs:OpenGLOpenGL|ESOpenVGX11R7.8blobDisplayserverlibwayland-clientWayland1.5framebufferDIXdriverhardware-specificbyShmuelCsabaOttoTraian;CC-BY-SA4.0intl;created2013-08-24;lastupdated2014-03-25KMSKernelModeSettingWaylandobsoletes2DdriversindisplayserverWaylandcompositorAPI:EGLAPI:EGLWindowmanagerKWinMutterWestonEnlightmentKWinCompizOpenBoxMetacityMutterIllustrationoftheLinuxgraphicsstack(byShmuelCsabaOttoTraian,CC-BY-SA4.0)Withtherequirementtorunwithoutanywindowingsystem,likeXorWayland,thisimpliesrelyingeitheron:Proprietarydevicedrivers(thebinaryblobatthebottom,left-handsideoftheillustation),Or,MesaTheMesa3DGraphicsLibrary,isanopensourcesoftwareimplementationoftheOpenGLspecifications.Theprojectalsohoststhedevelopmentofopen-sourcedevicedriversforgraphicchipsets.open-sourcedevicedrivers,usingtheDRM/KMSLinuxkernelsub-system,toimplementmode-settingandpage-flippingoperations.Whilethelateroptionrequiresmoredevelopmentwork,itpromisestoworkacrosstherangeofGPUsthathaveaMesadriveravailable. Italsobringsthea-prioribenefitsoftheopen-sourcemodel,withcommunityandfreelyaccessibledocumentation.NowthattheLinuxstackisclearer,let’scontinueonthatmission,andfindoutwhatdriversexistfortheRaspberryPi…TheRapsberryPiTheRaspberryPi3BroadcomBCM2837SoCincludestheVideoCoreIVGPU,whichcouldinitiallybeusedwiththecorrespondingproprietarydriver,andclosed-sourceimplementationofthegraphicslibraries. Whilesomeportionsofthatstackwasreleasedasopen-sourcein2012,mostoftheworkisstilldoneintheclosed-sourceruntimelibrariesandGPUcode,asdepictedinthisdiagram:applicationOpenGLESapplicationapplicationMediaOpenMaxOpenVGEGLKerneldriverVideocoreIVGPUOpensourceClosedsourceBinaryblobARM3D2DTheVideoCoreIVGPUdriverstack(CCBY-SA3.0)TheCheaderfilesandlibrariesfortheseBroadcomspecificimplementationsarelocatedinthe/opt/vc/includeand/opt/vc/libdirectoriesTheseheaderfilesandlibrariescanbefoundathttps://github.com/raspberrypi/firmware..In2014,BroadcomandtheRaspberryPiFoundationannouncedthedocumentationreleasefortheVideoCoreIV3DgraphicsprocessorTheArchitectureReferenceGuidefortheBroadcomVideoCoreIVGPUisavailableathttps://docs.broadcom.com/docs/12358545.,aswellasthesourcereleaseofthegraphicsstackunderaBSDlicenseThesourcecodefortheuserlandlibrariescanbefoundathttps://github.com/raspberrypi/userland.. Fewmonthsaftertheannouncement,thesourcecodeofaGallium-basedMesaOpenGLdriverfortheBroadcomSoCGPU,writtenfromscratchbyEricAnholt,wascommittedtotheMesaprojectThesourcecodefortheVC4drivercanbefoundinthesrc/gallium/drivers/vc4directoryoftheMesarepository.. Thispavedthewaytowardsopen-sourcedriversfortheRapsberryPiGPUsWhileEricAnholtmovedontootherresponsabilities,thedevelopmentofthesedriverscontinuesactively..TheRaspberryPi4BroadcomBCM2711SoC(formerlyBCM2838)nowincludestheVideoCoreVIGPU,that’sonlysupportedbyaMesadriverThesourcecodefortheV3Ddrivercanbefoundinthesrc/gallium/drivers/v3ddirectoryoftheMesarepository.. TheoriginalBroadcomproprietarydriver,specificallydesignedfortheBCM2837SoCGPU,doesnotworkontheRapsberryPi4. ThisMesaV3D(VideoCoreVI)driverconformstoOpenGLES3.1(asofMarch2021),whiletheVideoCoreVIGPUiscapableofOpenGLES3.2.Afterthisresearchphase,anditsfewhistoricalfindings,I’mconvincedthewayforwardtosucceedinmymission,istorelyontheseopen-sourcedrivers…TheProgrammingWiththesedrivers,runningOpenGLorOpenGLES,withoutX11,ispossibleusingtheDRM/KMSLinuxkernelsub-systemTheLinuxGPUDriverDeveloper’sGuideprovidesanextensivedocumentationoftheDRM/KMSsub-system.,incombinationwiththeMesaGenericBufferManagement(GBM)library.Lukily,Istumbleduponkmscube,whichisanexampleapplication,writteninC,thatdemonstrateshowtousetheKMS/GBM/EGLAPIstodrivebaremetalgraphics,andprovidesanimplementationofthemode-settingandpage-flippingoperations.Thebasicideaistousetwotriangles,coveringtheentirescreen,thatarerasterizedbysamplingtheshaderforeverypixel. Sothatleftmewith:1)LoadingacopyoftheShadertoyshaderfromthefilesystem://ThetemplateusedtoinputuniformsthatareautomaticallyaddedbyShadertoy, //andtocalltheShadertoyshadermainmethodentrypoint. staticconstchar*shadertoy_fs_tmpl= "precisionmediumpfloat;\n" "uniformvec3iResolution;//viewportresolution(inpixels)\n" "uniformfloatiTime;//shaderplaybacktime(inseconds)\n" "uniformintiFrame;//currentframenumber\n" "\n" "%s\n" "\n" "voidmain()\n" "{\n" "mainImage(gl_FragColor,gl_FragCoord.xy);\n" "}\n"; //CreatesthefragmentshaderfromalocalcopyoftheShadertoyshader staticchar*load_shader(constchar*file){ structstatstatbuf; char*frag; intfd,ret; fd=open(file,0); if(fd<0){ err(fd,"couldnotopen'%s'",file); } ret=fstat(fd,&statbuf); if(ret<0){ err(ret,"couldnotstat'%s'",file); } constchar*text=mmap(NULL,statbuf.st_size,PROT_READ,MAP_PRIVATE,fd,0); asprintf(&frag,shadertoy_fs_tmpl,text); returnfrag; }2)CompilingtheshadersontheGPU,creatingtheOpenGLprogram,linkingit,andinitializinguniformvariablesandbuffers://Holdstheplaybacktimeandframenumberuniformlocations GLintiTime,iFrame; //Thetriangles,toberasterizedbysamplingtheshaderforeverypixelofthescreen. //QuadsarenotsupportedbyOpenGLES,sowehavetousetwotriangles. staticconstGLfloatvertices[]={ //Firsttriangle: 1.0f,1.0f, -1.0f,1.0f, -1.0f,-1.0f, //Secondtriangle: -1.0f,-1.0f, 1.0f,-1.0f, 1.0f,1.0f, }; //Thevertexshader,responsibletopositionthegeometry. //Wesimplyneedtheidentityinourcase. staticconstchar*shadertoy_vs= "attributevec3position;\n" "voidmain()\n" "{\n" "gl_Position=vec4(position,1.0);\n" "}\n"; intinit_shadertoy(conststructgbm*gbm,structegl*egl,constchar*file){ intret; char*shadertoy_fs; GLuintprogram,vbo; GLintiResolution; //LoadstheShadertoyshaderfromthefilesystem,andcreatesthefragmentshader shadertoy_fs=load_shader(file); //Compilesthefragmentandvertexshaders,andattachesthemthereturnedprogram ret=create_program(shadertoy_vs,shadertoy_fs); if(ret){ printf("failedtocreateprogram\n"); return-1; } program=ret; //Linkstheprogram ret=link_program(program); if(ret){ printf("failedtolinkprogram\n"); return-1; } //Matchestheviewportwidthandheighttothescreenresolution glViewport(0,0,gbm->width,gbm->height); glUseProgram(program); //Initializestheuniformvariables iTime=glGetUniformLocation(program,"iTime"); iFrame=glGetUniformLocation(program,"iFrame"); iResolution=glGetUniformLocation(program,"iResolution"); glUniform3f(iResolution,gbm->width,gbm->height,0); //Initializestheverticesbufferthatholdsthetrianglesdata glGenBuffers(1,&vbo); glBindBuffer(GL_ARRAY_BUFFER,vbo); glBufferData(GL_ARRAY_BUFFER,sizeof(vertices),0,GL_STATIC_DRAW); glBufferSubData(GL_ARRAY_BUFFER,0,sizeof(vertices),&vertices[0]); glVertexAttribPointer(0,2,GL_FLOAT,GL_FALSE,0,(constGLvoid*)(intptr_t)0); glEnableVertexAttribArray(0); //Providestherenderingmethodtobecalledforeachframe egl->draw=draw_shadertoy; return0; }3)Finally,rasterizingthetriangles,sothateverypixelfromthescreengetssampledfromthefragmentshader://Thisiscalledtorendereveryframe staticvoiddraw_shadertoy(uint64_tstart_time,unsignedframe){ //Setstheplaybacktimeuniforminseconds glUniform1f(iTime,(get_time_ns()-start_time)/(double)NSEC_PER_SEC); //Setsthecurrentframenumberuniform glUniform1ui(iFrame,frame); //Rendersthetriangles glDrawArrays(GL_TRIANGLES,0,6); }Thecompletesourcecodeisavailableathttps://github.com/astefanutti/kms-glsl.TheFun AselectionofshadersfromShadertoy,thatrunsuccessfullyontheRapsberryPi.Youcanfindcopiesoftheseintheexamplesdirectoryoftheprojectrepository.I’vesuccessfullyrunshadersontheRPi3B+andRPi4,withRaspberryPiOSLite2020-12-02,Linuxkernel5.4.79.YoucanrunthefollowinginstructionstobuildtheCLIbinary:$sudoaptupdate #Installthebuildtools $sudoaptinstallgccmake #InstalltherequiredDRM,GBM,EGLandOpenGLESAPIheaders $sudoaptinstalllibdrm-devlibgbm-devlibegl-devlibgles2-mesa-dev #Clonetherepository $gitclonehttps://github.com/astefanutti/kms-glsl.git&&cdkms-glsl #BuildtheglslCLIbinary $makeTheVC4/V3Ddriverkernelmodulemustbeactivated. Assumingyou’veinstalledRaspberryPiOS,thiscanbeachievedbyrunningthefollowingcommands:1)Editthe/boot/config.txtfile,e.g.:$sudovi/boot/config.txt2)Setthefollowingproperties:#Required:Enablethefirmware/fakeDRM/KMSVC4/V3Ddriver dtoverlay=vc4-fkms-v3d #Optional:IncreasethememoryreservedfortheGPU #16MBdisablescertainGPUfeatures gpu_mem=64 #Optional:AvoidGPUdown-clockingbelow500MHzthatslowsFPSdown #Shouldbesetto250ontheRPi3 v3d_freq_min=5003)RebootyourRaspberryPi,sothatthechangesaretakenintoaccount,e.g.:$sudorebootYoucanthenrunshadersfromtheexamplesdirectory,e.g.:$./glslexamples/stripey_torus_interior.glsl OpenGLES2.xinformation: version:"OpenGLES3.1Mesa19.3.2" shadinglanguageversion:"OpenGLESGLSLES3.10" vendor:"Broadcom" renderer:"V3D4.2"Andcheckthatrenderer:"V3D4.2"ispresentintheconsoleoutput,toconfirmit’ssetupcorrectly.SoonafterIstartedtesting,Irealizedtheframeratewasfluctuating,andtheV3DGPUfrequencywasdroppingwellbelow500MHz. Thiscanbeobservedbyrunningthefollowingcommandfromaseparateterminal:$watch-n1vcgencmdmeasure_clockv3dThisissuehasbeenreportedinraspberrypi/linux#3935. ItseemsthedefaultgovernorscalestheGPUfrequencydown,despiteinstructionsbeingscheduledintotheGPUworkloadqueue. AsolutiontopreventGPUfrequencydown-scaling,istosettheminimumfrequency,byaddingv3d_freq_min=500tothe/boot/config.txtfileThisisnotoptimal,asitincreasespowerconsumption,whiletheGPUisidle..ThiscanalsobeusedtooverclocktheGPU. I’vesuccessfullytestedoverclockingtheV3DGPUto600MHz,whichresultsinanoticeableFPSimprovement.TheFutureThevc4-fkms-v3ddriverisknownasthefake/firmwareDRM/KMSdriver,wherethekerneldriverstilldelegatestheinteractionswiththedisplaycontrollertothefirmware. Anewervc4-kms-v3ddriver,knownasthefullDRM/KMSdriver,isnowavailable,wherethekerneldrivesthedisplaycontrollerdirectly.Igaveitatry,afteranupgradetothelatestkernelversionavailable(asofMarch2021):$sudoaptfull-upgrade $uname-a Linuxmaster5.10.17-v7l+#1403SMPMonFeb2211:33:35GMT2021armv7lGNU/LinuxUnfortunately,Ifacedtheissuereportedinraspberrypi/linux#4020. Iplantotryitagain,onceit’sfixed. ItseemsitmaybepossibletouseitincombinationwiththetouchscreendriverfortheDSIdisplay. Soitcouldpossiblyenableinteractivity,byfeedingthemouseuniform,withtouchscreeneventsfromtslib.Therearealsofewthingsthat,Ithink,wouldbelogicaladditions:ForwardkeyboardeventstotheinputkeyboardtextureFeedaudio/videoinputstomicrophone/cameratexturesPackageShadertoydefaultmediafilesLoadarbitraryinputtexturefiles…Whatstartedasatoyproject,toendyear2020light-heartedly,turnedouttobeasmall,yetveryrewarding,journeyintotheworldofopen-sourceGPUprogramming. IthinkIcansafelysayit:missionaccomplished!I’dbehappytohearyourfeedbackathttps://github.com/astefanutti/kms-glsl!