An investigation into the deep learning approach in ... - PLOS

Sentiment analysis is a branch of natural language analytics that aims ... deep learning approach in sentimental analysis using graph-based ... BrowseSubjectAreas ? ClickthroughthePLOStaxonomytofindarticlesinyourfield. FormoreinformationaboutPLOSSubjectAreas,click here. Article Authors Metrics Comments MediaCoverage ReaderComments Figures Figures AbstractSentimentanalysisisabranchofnaturallanguageanalyticsthataimstocorrelatewhatisexpressedwhichcomesnormallywithinunstructuredformatwithwhatisbelievedandlearnt.Severalattemptshavetriedtoaddressthisgap(i.e.,NaiveBayes,RNN,LSTM,wordembedding,etc.),eventhoughthedeeplearningmodelsachievedhighperformance,theirgenerativeprocessremainsa“black-box”andnotfullydisclosedduetothehighdimensionalfeatureandthenon-deterministicweightsassignment.Meanwhile,graphsarebecomingmorepopularwhenmodelingcomplexsystemswhilebeingtraceableandunderstood.Here,werevealthatagoodtrade-offtransparencyandefficiencycouldbeachievedwithaDeepNeuralNetworkbyexploringtheCreditAssignmentPathstheory.Tothisend,weproposeanovelalgorithmwhichalleviatesthefeatures’extractionmechanismandattributesanimportancelevelofselectedneuronsbyapplyingadeterministicedge/nodeembeddingswithattentionscoresontheinputunitandbackwardpathrespectively.WeexperimentontheTwitterHealthNewsdatasetwerethemodelhasbeenextendedtoapproachdifferentapproximations(tweet/aspectandtweets’sourcelevels,frequency,polarity/subjectivity),itwasalsotransparentandtraceable.Moreover,resultsofcomparingwithfourrecentmodelsonsamedatacorpusfortweetsanalysisshowedarapidconvergencewithanoverallaccuracyof≈83%and94%ofcorrectlyidentifiedtruepositivesentiments.Therefore,weightscanbeideallyassignedtospecificactivefeaturesbyfollowingtheproposedmethod.Asoppositetoothercomparedworks,theinferredfeaturesareconditionedthroughtheusers’preferences(i.e.,frequencydegree)andviatheactivation’sderivatives(i.e.,rejectfeatureifnotscored).FuturedirectionwilladdresstheinductiveaspectofgraphembeddingstoincludedynamicgraphstructuresandexpandthemodelresiliencybyconsideringotherdatasetslikeSemEvaltask7,covid-19tweets,etc. Citation:KentourM,LuJ(2021)Aninvestigationintothedeeplearningapproachinsentimentalanalysisusinggraph-basedtheories.PLoSONE16(12): e0260761. https://doi.org/10.1371/journal.pone.0260761Editor:ThippaReddyGadekallu,VelloreInstituteofTechnology:VITUniversity,INDIAReceived:May7,2021;Accepted:November16,2021;Published:December2,2021Copyright:©2021Kentour,Lu.ThisisanopenaccessarticledistributedunderthetermsoftheCreativeCommonsAttributionLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,providedtheoriginalauthorandsourcearecredited.DataAvailability:AllHealthNewsinTwitterDataSetfilesareavailablefromtheUCIMachineLearningrepositorydatabase(Indexof/ml/machine-learning-databases/00438),URL:https://archive.ics.uci.edu/ml/datasets/Health+News+in+Twitter.Funding:Theauthor(s)receivednospecificfundingforthiswork.Competinginterests:Theauthorshavedeclaredthatnocompetinginterestsexist. IntroductionDuetothetremendouscoveringandstandardizationofsocialmediaandInternetofThingsonourdailylife[1,2]peoplefeelmoreconfidenttoconsiderthisdigitalconnectedworldasanewcommunicationtool.ResearchinMachineLearning(ML)haswidelyaddresseddifferentwaystoassesspeople’sthoughtsandretrievemeaningfulcorrelationstobestquantifythem,thisisknownasSentimentAnalysis(SA).Thelatterhasrevolutionizedseveraldomainsbyconsideringusers’understandingandfeedbackaboutspecifictopicstoimprovetheirtrustworthinessandthereforebenefitsbusinesses[3],thisincludes: Business:assessingcustomers’voices[4],marketresearchandanalytics[5](e.g.,e-business),reputationmanagement[6],etc. Technology:Recommendationsystems[7],robots’adaptation[8],assessingastronauts’mentalhealth[9],etc. Socialactions:Realworldevents’monitoring,smarttransport/cities[10],socialmediamonitoring(i.e.,racismdetection[11,12]),etc. Politic:peacefulgovernmentsolutions[13],clarifyingpoliticians’positions,opinionsinversionprediction[14],etc. Healthcare:approachingpeoplefromdifferentbackground/racesbyextractingcommonfeedbacksandcorrelations[15],retrievinginsightsinordertoimprovetreatments(e.g.,breastcancertreatmentexperience[16],braindata[17]hasbeenextractedtoinfercorrelationsamongnaïvespeakers,etc). MostoftheseworksperceivedSAasaclassificationtask(e.g.,SupportVectorMachine(SVM)[18],NaïveBayes(NB)[19],biasimpactonML[20],etc.).Inthissense,recentworkshaveshownpromisingoutcomesbyboostingtheperformanceofthesealgorithms.In[21],afeatureselectionmechanismhasbeenproposedandoutperformssomeclassicalselectiontechniques(e.g.,Term-frequency,Chi-square,etc.)byprovidingmorecontexttothefeature’ssizereductionratherthanfrequency(i.e.,dataspread,outputcorrelation,etc.). Despitesomepromisingclassifiers(e.g.,NBwith94.02%accuracy[22],SVMandNBwith90%and95%respectively[23],etc.)inthedomainslikehealthcareforinstance,itisknownthatdata(e.g.,Functionalrehabilitation)arehighlycorrelated[24]andnotequallydistributed[25].ThoselatterexclusionsrequiremorebetteranalyticframeworksthatmergesbothcomputationalpowerandacoveringknowledgeinordertoadjusttheSAtothemedicalfield.Inthissense,graphgenerationtechniquesareknownfortheirexpressivenessanddeepdataprocessing[26]whichgaveawaytoarecentanalysistechnologyknownasgraphembedding[27].ThelattertechniquehasbeensubjecttomanyMLimprovements(e.g.,reducinginputsizeandfeatureselectionforanaccuratetextclassification[22,23],etc.). LatesteffortsonDeepLearning(DL)havebeenshowinggoodfunctionapproximationsratherthantraditionalMLones[28]byusingadditionalcomponents(i.e.,thresholds,weights,activationfunctions,etc.);however,SAforhealthcareimpliesadeepinvestigationatseverallevels,thatwasjustifiedin[29]byusinganaccompaniedtextinvestigationalongwiththeConvolutionalNeuralNetwork(CNN)algorithm,whichmeansDLstilllacksanextensiblefeaturelearningmechanismtobestanswertheSAprocessasadvocated.Inthiswork,weinvestigateanewdeepneuralnetworkmethodforSAwhichbetterapproximatesthedifferentaspectsofSA(i.e.,polarity,subjectivity,frequencyofterms/tweetswithintext,etc.),thiscontributionistwo-fold:1)improvingthefeedforwardpathbyproposinganembeddingstrategyfortheinputunitwhichreducesthedatatrainingcomplexitywithinalow-dimensionalspace.2)increasingthebackwardpath’sprecisionbyscoringthefeaturesfollowingtheirimportance(i.e.,highfrequency,betteractivationfunctionapproximation,etc.),whichguaranteesarapidlearningsurgewithagoodperformance(i.e.,highaccuracy,F-score,etc.).Furthermore,themodelhasbeenshowntobetransparentandefficient. Theremainderofthispaperisorganizedasfollows:Section2liststheresearchquestionsandasetofrespectivehypotheseswhichemphasizethedevelopedaspectsofthisresearch.Ouraimsandobjectivesaredetailedinsection3.Section4presentstheliteraturereviewandthetheoreticalaspectofthisresearch.Whereas,ourproposedmethodsarepresentedinsection5,thisisfollowedbyanexperimentalstudyinsection6.Weevaluateourmodelinsection7,andthenwecriticallydiscussedthewholeworkinsection8.Section9concludesthepaperandgivesfewperspectives. MotivationThemechanismoftheactualDeepNeuralNetwork(DNN)hasbeenofficiallyproposedby[30]asasupervisedMulti-LayerPerceptron(MLP).Toourbestknowledge,thesameauthorswerethefirstintroducersofDNNstransparencybytrainingeachlayerindependentlyandlearningtheircorrelatedrepresentations.Thiswasafeed-forwardmodelofmultiplelayers(calledconnectedcomponents)ofnon-linearactivationfunctions.However,thetheoryoftheinput’sinfluenceontheoutputperformancewithinneuralnetworkswasdiscussedfewyearsbeforeby[31]knownastheproblemofCreditAssignmentPaths(CAPs).ThelatterconsistsofdecidingwhichDNNcomponentsareinfluencingthemodelperformance.Whilethisproblemcouldbeaddressedinadifferentmanner,similarworksagreedonthefinalperformanceasthemaincriteriatojustifythemodel’sefficiency.In[32],authorshavebeeninvestigatingthestabilityofDNN(i.e.,multidirectionalLSTM)componentsmodelledasagridasawaytostopDLmodelvanishingproblem.Althoughauthorsin[32]haveachievedstate-of-the-artperformance,thecomplexityoftheinputspaceandthestateactivationlayerin[32]remainsanissueifdeployedwithlimitedresources. Nowadays,withtheemergenceofNeuroscienceandartificialneuralnetworks[33],CAPsarenotonlylimitedtoacertainlayer.Moreover,back-propagationstrategy[34]remainsinefficientincertainvanishingoroverfittingproblems,whicharemorelikelytooccurduetotheequalconsiderationoftheinputsamples(see[21]). AsSAbecamepopularformanyDLapplications,thelackoftransparencyindecisionmakingwithinspecializeddomainslikemedicine[35]isquitemisleadingandsomepracticesmayopposetotheGeneralDataProtection(GDPR).Toourbestknowledge,CAPshasnotyetbeeninvestigatedinthisresearchareawhereasitwastheoriginofDLtransparencyasstatedbefore.Therefore,bythisresearch,weaimtoinvestigateCAPstheoryforatransparentDNNstructurethatbestanswerstheSA.IncontrasttotheDLmodelsfromliterature,wewanttokeepthecomplexity(i.e.,special/temporal,see“Complexityanalysis”)asloweraspossible,andthiswillbedonebyactingonthebuildingcyclesofaDNN(i.e.,feedforwardandbackwardpaths)whilerestrictingtheinputfeaturesinalowerspacerepresentationandthenscoringthederivativeinstanceswithaselectionmechanismrespectively. ResearchquestionsandhypothesesInordertobestunderstandtheproposedresearchinvestigationaswellastheobjectivemethod,thefollowingquestionslistedinTable1aimtoframethisresearchintotherightcontext.Asetofhypotheseshavebeenproposedfollowedeachresearchquestion. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable1.Theproposedresearchandthefollowinghypotheses. https://doi.org/10.1371/journal.pone.0260761.t001 AimsandobjectivesOnlyfewattemptshavetriedtoassociategraphtechnologiestothedeepsentimentanalysisprocess[37,38].TheaimoftheproposedmethodistostudytheinfluenceoftheinputnodesandhiddenlayersonthefinalDNNsoutputs,insuchway,gettingtherightsamplefeatureswillhelptoreducethefeaturesvectorspacewhilekeepingthemodelrationality.Thiswasinspiredfromtheattentionmechanism[39]alongwithdeployingthedeepneuralarchitecture.Thestudywillfocusonpeople’stweets,thegoalistoenrichtheDNNstructurewithgraphembeddinglearning[27],whichwillberefinedthroughaselectivestrategy.ThefollowingFig1associateseachproposedresearchquestionwiththeenvisagedaimsandobjectivesrespectively. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig1.Theproposedaimsandobjectives. https://doi.org/10.1371/journal.pone.0260761.g001AsshowninFig1,weaimforeachresearchquestiontobeansweredfollowingtheassociatedobjectives,andthatforthefollowingpurpose: AnsweringthatquestionwillhelptoemphasizetheincreasingtrendtowardexplainableDLandthedifferentapproaches(see“TransparencyinDL”). ExpendingthisquestionallowstofigureoutaconvenientwaytoabstractagivenDLproblemwhilebeingrationaltotheinternalstructure(see“Abstractionstrategy”). Byexploringthisquestion,mostrecentGNNshavebeenreviewedandthemainobstacleformakingthemunderstandablewashighlighted(see“Graphbasedneuralnetworks”). ThisquestionwillhelptorevealapartitioningmethodthatpermitstoidentifytheDNNsunitconcernedbytheproposedmethod(see“Methods”)andthathasimpactonthewholeperformance. ThisquestionwillmotivatethemostrecentattentionalmechanismwithinSAandthewaytomergethatwithgraphembeddingsmethods(see“DLapplicationsonSA”). LiteraturereviewInthissection,wereviewmostrecentapplicationsofDLonSAandtheirperformance.Then,weaddressexplanabilitywithinDLbyemphasizingrecentgraph-basedlearningmodels. Researchstrategy ThefollowingstrategydenotesthemainresourcesandthedataextractionschemewhichallowsagoodreflectionofthemultidimensionalitytopicofDNNswithrespecttotheSAfield.Thisisfollowedbyanevolutionchronologyandacarefulcombinationofthetopics’components(CAPs,graphs,SA,DL)whichtogethermotivatetheproposedmethod. Literatureresources.IEEXplore,ScienceDirectandSpringerresearchdatabaseswereinvokedinordertoretrievepapersfromjournalswhichrefertoexplainableDL,journalpapersreferringtoSAhavebeenreviewedfromPubMeddatabase,thishasbeenrefinedtoincludeworksbasedonDLinparticular.ThecontextandkeywordsrelatedtoeachdatabaseaswellastheselectionresultsareillustratedinFigs2and3respectively,whereasthefollowingdiagramsummarizestheselectionstrategy(Fig4). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig2.Researchdatabasesandkeywords. https://doi.org/10.1371/journal.pone.0260761.g002 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig3.Releasedpapersforeachdatabasecorrespondingtoeachrelatedsubject. https://doi.org/10.1371/journal.pone.0260761.g003 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig4.Journalpapersselectionmethod. https://doi.org/10.1371/journal.pone.0260761.g004 Subjectevolution.CAPsandexplainableDL.CAPsisahistoricalproblem[40]whichexplorescausalpathsstartingfromadjustinginput’sweightstoanoptimaloutput.ThemajorityofworksongraphexplainableDLhaveaddressedCAPsproblemfromspecificangles,usuallyreferredtoas“modelspecific”[41];however,onlyfewattemptshavetriedtopositionaDNNasacompositionalunit[42]andthebestwaytoassigninputvalueswhichreferstothehistoricCAPs.AsshowninFig5,CAPsisgainingmoreandmoreattentionduringlastyears,aswellaspublishedpaperswithareferencetoexplainableDL(XDL)andCAPs.Mostofthemwerebio-inspiredwhichtreatcreditsaselectricsignalscomingfromexternalsensors,knownas“cause-affect”strategy. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig5.PublishedpapersreferringtoCAPsandexplainableDLwithreferencetoCAPs. https://doi.org/10.1371/journal.pone.0260761.g005GraphsandCAPs.Asstatedbefore,researchonCAPshasbegunasawaytoassigncreditstobetterminimizetheerrorfunction[42].Fig6illustratesnewcategorizationofCAPs’approachesbasedonneuronpaths’detection. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig6.RecentapproachesforDNNscreditassignment. https://doi.org/10.1371/journal.pone.0260761.g006ThemainquestionwhichwaspreventingCAPsfrombeingwidelyexploredasanefficientperformanceparameterwas“whetherthebrainbackpropagatesornot”;inthissense,graphshavebeensubjectofresearchinordertorepresenttherelevancebetweendatapatterns[43],RNNshavebeenfirstlyproposedtodealwithbackpropagation,thenLSTMs[44,45]andSlicedRNNs(SRNNs)[46]foraconstantvanishingpreventionandlongtermdependenciesrespectively. AsshownbyFig7,newmodelsbecamepopular,they’reallcharacterizedbytheirgraphicnaturewhichnotonlytrytosolvealearningproblem,buttolearnhowtheresolutionisinferred[47].StochasticlearningGraphs(SGs)[48]forinstanceintroducesnewgradientsettingtobestreducetheloss. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig7.DNNsmodelsdistributionoveryearsasagraphbasedsolutiontoCAPs. https://doi.org/10.1371/journal.pone.0260761.g007Moreover,GenerativeAdversarialNetworks(GANs)havebeenprovingtheirefficiencyintransferablelearningbyrevealinggenericanalysispatterns[49].However,large“discrete”graphs(e.g.,Multi-hiddenDNN)duetodiscreteindependentweights.Furthermore,AttentionlayershaveextendedDNNstructure[39](AGs)withanimportancedegreeofnodesorlinkswhichalleviatethediscretelearningtobeinductivewithlesscomputation(i.e.,withoutmatrix-factorization). ReinforcementLearning(RL)wasthemosttargetedmodelwhiledealingwithCAPs,becausethewayneurons’weightsareupdated(byassigningafinalweighttoacertainneuron)isverysimilartotheconceptoffailure/rewardwithinRLfollowedbyseekinganexplanationfortheresult. Sentimentanalysis SAhasbecomingabasic-blockunitformanymodernplatforms;itsevolutionhasseenvariouschangesandappellations[50]alongwiththetechnologyandanalyticsusedfortheanalysis.Fig8representsaprogressbarofSAaccordingtoneuralnetworksevolution.DLhasrevolutionizedthewaySAisconducted,startingfromasingleperceptronthatonlysupportsalimitnumberofweightsandbias,toarelativelybetterapproximationoffunctionswithMulti-LayerPerceptron(MLP)andtheintroductionofback-propagationalgorithm.Bymid90s,SAbecameverypopularbytheintroductionofkernelfunctionsandHuman-interfacemachinesknownas“BrainComputerInterface”. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig8.BriefchronologyofSAfollowingthedevelopmentofDL. https://doi.org/10.1371/journal.pone.0260761.g008AscertainadmitthatemotiondetectionisthefuturetrendofSA[51],thelatterisstilldominatingthefieldofmedicineandpsychologywhereDLisplayingakeyroleontransformingpeople’sentimentsintocomputationalaspects. SentimentanalysisthroughCAPs.AsmodernSAprocessmayimplydealingwithlongtextframesandguaranteeinnerorouterdocumentdependency,thiswillinitiallyrefertoassigningcertaindocumentstopre-trainingstage;therefore,itcanbesubjectofCAPsinordertofigureouttherightparameters.Forourknowledge,thelatterproblemhasnotbeenaddressedfromaCAPsviewpointyet;However,asshownbyFig9,itwasremarkablyshownasimilarinterestonbothgraphembeddingandattentionmechanismswhichreflecttheeffectivenessofgraphsinthoseresearchareasintermsofselectivelyhighlightingtheactivesetofneuronswhichcanbeoptimizedandtheoneswhichmayimpactthepredictedsentimentinbothCAPsandSArespectively. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig9.Similarresearchaddressing“SA”and“CAPs”relativetographtechnologiesbetween2000–2021(basedonthepreviousanalysis(Fig7),graphshavebeengettingmoreattentionbyyear2000). https://doi.org/10.1371/journal.pone.0260761.g009 DLapplicationsonSA.SA[52]hasprovenitsabilitytoretrievehuman’sfeelingsfromseveralconfusingtexts.However,longtermdependencyisoneoftheDNNs’applicationlimitsonSA,whichconsistsofpreservingatraceableexecutionofthemodel[53].Asapossibleanswertothefirstpartof“Researchquestions”(RQ5),recentmodelsfromtheliterature(Table2)triedtoaddressthatissuebyhybridizingsomemodels,likeLSTMwithGCN[38]forinstance;however,amechanismthatdetectsimportantpatternsismuchmoreneededwithsourcevariantdatasets,notonlyforimprovingaccuracy,butforthelearningvisibility. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable2.WorksonDLforSA. https://doi.org/10.1371/journal.pone.0260761.t002 TransparencyinDL.TherehasbeenalotofresearchaboutclarifyingDNNsandwhetherunderstandingtheinternalconnectionofneuronscouldimprovethemodelperformance[69].ImagingisoneoftheemergingfieldsinDL,themajorityofworkstriedtoexplainimagingsystemsfromspecificproblems[70,71].However,languageprocessingaccompaniedwiththeavailabilityoflargetextdatasetbecamecentreofinteresttomanyresearchers,oneremarkableworkwasdoneby[72]forhugetextcorpusexplanation;althoughtheimagingsystemismoreclarifiedandflexible,thewaythegraphwasgenerateddoesn’tbenefitfromgraph-basedtechnologiesthatoptimizetheinputstartingfromnaivegeneration. Overall,explanabilityinDLcanbecategorizedinto: Example-basedapproaches;researchinthisareaisalwaysconductedthroughatraining-example,byspecifyingsomeinitialobservationswhichwillbeverifiedthroughfeatures’extraction,thisdisciplineiswidelyadapteddespitethedifficultyofverifyingthetrustworthinessofeachexample,thiscovers: ✓Gradientmethods(e.g.,Guided-backpropagation,Layer-wiserelevancepropagation[72]),whichaimtoabettergradientoptimization. ✓Saliency-featuremap[73]formeasuringpatternimportancewithinimagesandvideos. Model-basedapproaches,whichconcentrateontherawdata,they’reusuallyreferredtoasinputoptimizers.Somerecentworksincludethepre-processingstageofDARPA[74]wheretheexplainableinterfaceisbuiltonusers’psychologicalaspect.[75]haveexploredthefusionalaspectofDNNswhichaimsto“mimic”afunctionaggregatorusingfuzzynetwork,etc. Graphbasedneuralnetworks.Graphsareplayingacrucialroleinprocessingdataandpreservingtheirsemantics[76].TheideaofcombininggraphtechnologiesandDLisnotrecent[77].Asaproofofthat,manygraphmanipulationshavebeenintroduced:graph-pooling[78],graph-attentionnetworks[39],etc. However,fewattemptshavecoupledlabelledgraphgenerationwithadeeplearningmodelapartfromtheactivationfunction,whichmakesthemextremelyhardtoexplainortointerpret.Fig10comparesfewrecentworksongraphexplainableDL. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig10.Overallcomparisonofpredictiveaccuracy.EVCT[72]:ExplainableandVisualizingCNNfortextinformation.XGNN[73]:ExplainableGraphNeuralNetwork.STC-PG[75]:SpatialTemporalandCausal-ParseGraph.KGCN[76]:Graph-basedConvolutionalNetworkforchemicalstructure.HAGERec[77]:HierarchicalAttentionGraphConvolutionalNetworkIncorporatingKnowledgeGraphforExplainableRecommendation. https://doi.org/10.1371/journal.pone.0260761.g010Themainobstacleofabstractingeverysingleunitofadeepneuralnetwork(see“Abstractionstrategy”)asagraphstructureisthenon-compliancewithback-propagationprocess.Theworkdoneby[75]isaproofofthatwheretheyhadtocreateafunctionaggregatorthatsimulatesthetrueChoquet-integralmechanism,becausegraphscouldbeencodedasadjency-matrixforthebest;andthatdoesnotfitwiththeback-propagatorasafunctionoptimizer.AsananswertoResearchquestions(RQ3),weinvestigaterecentefforts(Fig10)andwithinthebelowsub-section,inordertoretrievecertainlimitsonGNNsandmotivateamodel-basedapproachontheinputunitoftheDNN. Analysisanddiscussionongraph-basedSA.Theconductedevaluationillustratedby(Fig11)depictsmostDLstructuresandtheirvariationsintermsofaccuracyfollowingeachanalysislevel(see11).Whenconsideringdocumentsasawhole,LSTM-basedapproacheswerecrucialandshowedgoodperformancetocaptureinter/intradocuments’correlations.However,aslongaswemovefurtherfromsentence-basedtoasingleaspectlevel,thereismuchinterestonaspectsembeddingwithattentionnetworks,thelatterwereabletogatherneighbourhoodcontextforbettersentimentclassification.Thatcouldbenoticedinarecentmulti-modaltrends’analysis[67],whereRNNandLSTMfailtocaptureemotions’boundaryforthewholevideowhileAttention-basedCNNshowedgoodperformance(seeTable2). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig11.MostusedDLmodelsinSAandtheiraccuracy. https://doi.org/10.1371/journal.pone.0260761.g011Thefollowingnotesexpressfewlimitsofrecentworksonthisarea: GNNs(e.g.,Graphattentionnetworks,Attentiongraphs,Stochasticgraphs)(Fig7)arewidelyconsideredintheareaofconnecteddata,butlargelabelledgraphsstillrepresentanissueduetotheirexponentialgrowth,thereforemovingfromhighdimensionalitytolowspacerepresentationisconditionedbybeingdiscriminativetotherawdataparameters. TransferablelearningwhichconsistsofgeneralizingtheDLmodelfromaspecificobservationtootherdomainsstillanissuetomanyDLmodels,becausetheyarebuiltonaspecificdataset(s).However,asjustifiedby[79,80]afurtherapproachcouldbeperformedbysettingupaninputmechanismthatmapthecomplexityofrawdatatosmallerframeswhilebeingexpressive. Highdimensionalfeatureanalysisremainsanissueformostdependency-basedmodels(LSTM[80],GRU[59]);somesolutionshavebeendeployedlikeskipdataconnections[81]toreducetheinputsize,theymaypreventsomevanishingcases,buttheyaddmorecomplexityasadditionalhiddenlayerstothegradient.Thisiswhymajorityofresearchisnowturningtoaddresstheagnosticaspectoftheexplanation,inordertoimposeastandardlimitfortheinput. Thepreviousargumentationsfallintotheexample-basedapproach(see17),whereamodelselectionstartsfromanobservedfact,likeneighbourhoodaggregation,shorttermdependency,etc.However,thesemethodsneglecttheimpactofDLinputunitsontheperformance,thingthatjustifiesthe“accuracy”paradoxes(Fig11)eventhoughasentenceoranaspectmayreflectasimilarsentiment.Therefore,thechallengewillbetoprovideanexplainablesolutiontotheDNNinputunit(i.e.,model-basedapproach(see“TransparencyinDL”))asananswertothe“Researchquestions”(RQ1),whichsatisfiestheCAPs(Fig9),andthisisbasedonthecurrentresearchtrend(Fig7). MethodsAsthehealthcaredomainisknowntobecriticandfullofcomplicatedscenariosthatdonotforgivemistakes,oneaccuratewaytoperformadeeplearningtechniqueisbypreservingthemodelrationality[82].Althoughmodeloriented[83]andexample-basedapproaches[84]haveshownanexplainableindependencylevelandaninputdependentoptimizationrespectively,theybothpositiontheproblemofclarifyingDNNswithinabarrierofhighinterpretabilitybutlowaccuracy,andviceversa.TheproposedapproachinthispaperconsistsofdesigninganovelDNNbasedonahybridgraphembeddings/attentionscoring. DNNsareknowntoprovidehighaccurateoutcomes,thisisknownasthemodelperformance.Formallyitisdescribedas: Nisthenumberofinputandhiddenlayers disthedesiredoutputandzistheactualoutput Mathematically,theoutputgeneration(z)throughthefeed-forwardandback-propagationcyclesisexpressedasaserieofpartialderivatives[33].Forinstance,supposethefollowingin-depthviewofadeepneuralarchitecture(Fig12)whichiscomposedoftwohiddenlayers,twoinputs(XA,XB)andtwooutputs(ZA,ZB). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig12.TwohiddenlayerDNNstructure. https://doi.org/10.1371/journal.pone.0260761.g012 Abstractionstrategy Inordertoanswerresearchquestion(RQ2)(see“Researchquestions”)andfollowingthestructuredepictedbyFig12,wewillexploretheimpactoftheperformance“P”ontheinternalDNNstructure.Byconsideringbothweights“w1”and“w3”,thiscouldbeexpressedbythechainrule(1)and(2).ThepurposeistojustifyastructuralunitoftheDNNmodelthatcouldbeoptimizedwithcompliancetotheDNNfeedforwardandbackwardpaths,see(Researchquestions(RQ4)). (1)(2) Itisnoticeablethattheselectedpartialderivativeunitsareequalwithrespecttoboth“w1”and“w3”andthiswillbethesamefortheunitswithrespectto“w2”and“w4”.Thatreferstotherepetitiveunit(Fig12),whichmeansithasnodirectimpactontheglobalperformanceasoppositetothedecisionalunit,where: thelastmultiplayerY1⊗w5givesq1asaninputtowardtheactivationfunctionandgeneratesZaasbothPath1orPath3. However,itisobservedthatY1isalsoimpliedtogenerateZbbutthistimefromthemultiplayerY1⊗w7andgivesq2tothesecondactivationfunctionwhichformsPath2orPath4. So,asmuchaswemovefurthertotheinput,therearemorecomputationalunitswhicharereused. Problem. BothInputs“Xa”and“Xb”participateforanintermediatecomponent“Y1”whichhasanimpactonthefinalmodelperformance. Findawaytoestablishanimportancedegreebetweenmodelinputs(e.g.,“Xa”and“Xb”)tofigureouttheone(s)withhigherimpactonthefinaloutput. Inputspaceembedding Embeddingsongraphsareknowntobeveryusefulindealingwithhugegraphdataandrandomdistribution[85].SupposeG(N,E)agraphofNnodesandEedges,where:E∈[1…m]andN∈[1…n]. Themappingfunctionisbasedonathresholdwhichanalysestheneighbourhoodconnectionsofeachnode,suppose(n=500)isamaximumallowedconnection: Incaseofnodeembeddings,foranoden1withc1connections: Map={N},f1∈Nandc1<=500; orMap={N—f1}wherec1>500. TheproposedmodeldepictedbyFig13,consistsofagraph-basedstrategywhichaimstoreducetheinputrepetitiveunitintoalow-levelspacerepresentation,thenintoasmallvectorunitwhichmayalleviatethecomputationcomplexityofthewholeDNNmodel. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig13.TheproposedmodeforSA. https://doi.org/10.1371/journal.pone.0260761.g013 Features’selectionviaattentionscoring Insteadofmovingfromtheembeddedvectorspace(see[23])throughtheactivationfunctions,ithasbeenconsideredtoscoretheembeddedfeatures(v1…vn)followingeachhiddenlayer(L1…Lk)withasetofweightsaw,w=[1..n]. (3)Thescorevectorrepresentsatraceofreachingfeatures,thelatterwillbemainlyenvisagedbytheback-propagationlossfunctionoptimizer(seealgorithmbelow),thereforebyconsideringtheactivationfunction((4)isthe”SoftMax”forinstance),theattentionweightaw(i)forahiddenlayer(t)willbecalculatedasfollowing: (4) Startingfromtheembeddeddistributionoffeatures,the“Gaussian”distancemetric[86]hasbeenconsideredtoscoresimilar(close)featuresandthereforetogeneratea“decorated”neuralpaththroughthe“SoftMax”functionforinstanceandrepeatedlytoachievebestdistribution.Alevelofgenericityisaimedtobereassuredintermsoftheactivationfunctionselectionaswellastheembeddedfeaturevector.Tosummarize,thecorrespondinglearningalgorithmwillbe: Algorithm:ToimplementtheproposedDNNmode(Embeddingandscoring) 1.Input:.txtfiles//rawdataset 2.Output:sentiment-polarity 3.ProcedureSA 4.Graph_SA=Networkx_Upload(pathtothecsv_file) 5.SamplesInitializing 6.vect=Embedding(Graph_SA)/*thiscallmaybenode/edgeembedding*/ 7.    FOReachfeaturewithinvectdo 8.        Input[x]=feature 9.        FORallxinDNNdo 10.            Output[x]=module.forwardPropagation(Input[x]) 11.            IFOutput[x]>=threshold/*thresholdcouldbemaximumnodeconnectivity(e.g.,mostfrequentaspects*/ 12.                Scored[x]=Output[x]//theselectedfeature 13.          End 14.         Input[x+1]=Output[x] 15.        End     /*Activationfunctioncondition(e.g.,Positivesentimentpolarityandattentionweightscalculation(2)*/ 16.    Sentiment-polarity=condition(Scored) 17.    IFstilltrainingthen 18.        FOReach[k-x]ScoredfeatureinDNNdo//kisthetotalfeatures’number 19.            Scored=module.BackwardPropagation/*Backpropagationwillstopiffeatureisnotscored*/ 20.            Input[x+1]=Scored[x] 21.        End 22.    End End Thealgorithmabovecanbeexplainedinthreemainparts: Thegraphgenerationandtheembeddedvectorextraction(see“InputSpaceEmbedding”),thiscoversline1tothe10thofthealgorithm.Theforwardactivationfunctionisappliedforeachembeddedfeature. Theconditionalstepwhichisvariantaccordingtoaspecificdomain(e.g.,mostfrequentfeatureinourcase),thiscorrespondstotheline11. Thefeatures’scoring,whichaconditionalstepaswell.However,itdiffersfromthepreviousoneaseachfeatureisconditionedwiththeactivationfunctions’requirements(i.e.,approximation,limitvalues,polarity,etc.). Solutionforhighdimensionalspace Ourproposedmode(checkthenumberofmodelswithnamesofeachmode)focusesontheinputunitoftheDNN,whereithasbeenshownthroughthechainrule(1)and(2)thatanyinputstream(Fig12)followsaspecificdecisionalpathwithrespecttothefeatures’weights.Ourcasestudy(see“Experiments”)imposesa2-ddimensionalrepresentationwhichcorrespondstothe“station-polarity”prediction.Thishasbeenachievedthroughagraphgenerationwithaneighbourhoodembeddings.Therefore,mostinfluentialnodeswithinagivenstationaretheoneshavingminimalGaussiandistance(i.e.,polarityofthemostfrequenttermwithinthetext.). However,certainDLtasksliketimeseries[87],adversarialexamples[88]requireanextensionoftheclassicalclosenessmethods(i.e.,Gaussiandistance),asthedatamaybedistributedwithink-dimensionalspace.Followingthegraphembeddingsstrategydenotedpreviously,asolutiontothemultidimensionalspacemustsatisfyanumberofcriteria: Theresultingembeddedstructuremustshowareducedfeaturesamplethantheoriginalinputone. Theembeddingfunctionmustcomplywiththeactivationfunctioninordertocopewiththepathdecoration. Asimilarprocess(i.e.,embeddingsandscoring)needstobeensuredwithinthek-dimensionalspaceinordertopreservetheoutputsemantic. Theprojectionoftheabovecriteriaresultsonthemappingprobability[89]ofafeature’sinstancexiinalayeriwithitsrespectivepatternxjonalayerj.AhigherprobabilityPi|jmeansacloserinstanceifromj(i.e.,station-polarityinourcase): (5) Therefore,byconsideringallthek-dimensionalspace,thescoringfunction(3)aswellastheactivationfunction(4),theoutputattentionweightaw(i)foralayer(t)willbegivenby: (6) Thereisaclearmatchbetweentheresultingscoringfunction(6)andtheactivationfunction(i.e.,SoftMaxforinstance),andthatconfirmsthesecondpartof“Researchquestions”(RQ5)onthecomplianceofthefeedforwardpathwiththebackwardone,whichenablesanefficientperformance(see“ImprovingDNNperformanceviaadeterministicbackwardwalk”). ExperimentsInthissection,anumberofempiricalexperimentshavebeenappliedontweetsHN-datasets(see27),datahasbeencollectedandunifiedfrom16differenthealthnewssources(stations),theproposedSAmodelgoesbeyondpolaritydetectionofpeople’sfeedbacktothemostinfluentialaspectsandsentenceswhichcontributetopolarityandsubjectivityvariations. Afterdatahasbeencleanedandpre-processed,weaimtobuildapredictiveanalysisaroundmostinfluentialtokensamongtweets,afterthatweshowtheroleofedgeembeddingintermsoftransparencyandthebenefitofvisualizingthepolaritydistributiononareducedplan. Datasets Healthnewstweetsdatasets(HN-datasets)[90]consistsof16differentsourcesofpeople’stweetshavingexperiencedorhavebeenexposedtohealthcaresituation.Datasourcesarerepresentedthroughdifferenttextfiles(i.e.,goodhealth.txt,foxnewshealth.txt,cnnhealth.txt,etc.),whichcontainmorethan58000instancesand25000attributes.ThefollowingTable3listssomefeaturesof“KaiserHealthnews”,“Foxnews”and“GoodHealth”stationsforinstance. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable3.Characteristicsofthreehealthtweetsdatasets. https://doi.org/10.1371/journal.pone.0260761.t003Thesedatasetsareusedtoprovethemodelworkingstrategy.Ithasbeendecidedtousethesedatasetstodealwithheterogeneousdata(i.e.,differentencoding,insignificantwords,healthcaredomainspecifications)andperformaglobalSAoftweets. Developmentenvironment ThisworkhasbeendoneonaUNIXsystem(UbuntuKylinver.20.10,architecturex86_64,processorintelcorei5).Python3.8wasthemainprogramminglanguageadoptedforimplementingthedataproceduresandthefollowingdataanalysistasks(seenextsub-sectionsinthecurrentsection“Experiments”).JupyterwasthemaindevelopmentAPIwithsomeofthefollowingpythonlibrariesforbasicfunctionsandvisualizations: The“glob”moduleasaUnixpathnamestylefordatasetsuploading. “nltk”asanaturallanguagetoolkitforstopwordsremoverforinstance. “re”moduletodealwiththeunstructuredtweets’filesasregularexpressions. “math”librarytoinvokemathematicalfunctions(e.g.,“Tanh”,“exp”functionstoimplementtheDNNactivations,“log”functionforlosssimulation,etc.). “WorldCloud”libraryforfrequenttokensdisplay. “Networkx”forgraphgeneration,etc. Datacleaningandpre-processing Thechallengingaspectaboutretrievingtweetsfromdifferentsourcesistheheterogeneousnatureofdatathatconsistsofdifferentencodingstyles(utf-8,cp1252,etc.,seeTable3),becauseanoverallSAaroundspecificdatasourcesisaimedtobeachieved. Textsplit.Astweetsaretotallyinformal,alistofspecialcharacters[。

?、~@#¥%……&*();:\s+\.\!\/_,$%^*(+\"\’]+|[+―!]hasbeenconsideredtosplitlinesintorawsequencesoftweetscontainingonlynaturallanguageterms. Stopwordremover.Tweetswithintheabovedatasetcomewithunstructuredtextualformat,thereforeapropertweetsanalysisconsistsofsplittingsentences/aspectsandremovingallsortofnon-significanceinordertoretrievethemostmeaningfulsentiment.NLTK’sstoplistEnglishwordshasbeenusedwithmoredomainspecificnon-relevantwords(i.e.,new,may,com,etc.). Statisticalsentimentanalysis Insteadofmeasuringindependentwordcombinations[91],theproposedapproachaimstoachieveaglobalsentimentpolarityofthewholedatacorpuswhichmergessources’heterogeneity,globaltermrelevantfrequencyandanadditionalsentimentfeaturecalled“subjectivity”.Aword-clouddistributionofmostfrequentwordsrelatedtohealthcarewithin“everydayhealth”,“gdnhealthcare”,“usnewshealth”isdepictedbyFigs14–16respectively. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig14.“evcerydayhealth”top-10. https://doi.org/10.1371/journal.pone.0260761.g014 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig15.“gdnhealthcare”top-10. https://doi.org/10.1371/journal.pone.0260761.g015 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig16.“usnewshealth”top-10. https://doi.org/10.1371/journal.pone.0260761.g016 Polarityvssubjectivity.Inhealthcaredomain,itiscommonlyusedtodetachthesentimentpolarityfromthesentimentsubjectivity[52,91,92].However,asillustratedbyFig17,ithasbeenfoundahighcorrelationbetweenhighfrequenttokensandtheircorrespondentpolarity/subjectivity.ThePolar{P}andsubjective{S}valuesareinterpretedasfollows: Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig17.Overallpolaritydistribution. https://doi.org/10.1371/journal.pone.0260761.g017P={>0→Positivesentiment                 0→Neutralsentiment                 <0Negativesentiment} S={0→Objectivesentiment         >0→Subjectivesentiment} Figs17and18showtheoverallpolaritydistributionaswellaspolar/subjectivevariationsrespectivelyofhealthnewstweetsbasedonrelevanttermsfrequencydistribution. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig18.Subjectivityandpolarityoftweets. https://doi.org/10.1371/journal.pone.0260761.g018Amongthe16-healthnews,only34.3%offrequenttweetsexpressednegativehealthcaresentiments(P<0),while70.4%ofthemwereobjective(S<0.5),thisisduetotheinformalnatureoftweets.Furthermore,aninterestingobservationconcernsmostfrequentterms(Figs19and20)wheretherewasaparallelsymmetricdecreaseofsentimentstowardsnegativeandobjectivefeedbacks,whichimbalancestheoverallpositivityoftweetsaswellastheirsubjectivity. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig19.Termsfrequencyandpolarity/subjectivity. https://doi.org/10.1371/journal.pone.0260761.g019 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig20.3-dplotfrequency,polarityandsubjectivitydistribution. https://doi.org/10.1371/journal.pone.0260761.g020 Predictiveanalysis Bytheproposedmodel,itisaimedtogobeyondthesubjectivityorpolaritydetection,toachieveatransparentpredictiveanalysisoftweets.Thegoalistotaketheaboveobservationsovertweetslevel,buttothedatasourcelevel.Thetechniqueconsistsofagraphgenerationwhichiscentredaroundthe16healthnewsstations,sogivenasourceoftweets,itwouldbepossibletopredictthesentimentpolarity/subjectivityinsteadofgoingthrougheachtweet,thentogetherthesestationsareconnectedwithinamap(Figs21and22).Thisapplicationcouldbeseenascommunitysentimentpolarprediction.Thefollowingdefinitionshavebeenproposedtobetterapproachthe“Researchquestions”(RQ3andRQ5). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig21.Station-polaritygraphgenerationwithoutedgeembedding. https://doi.org/10.1371/journal.pone.0260761.g021 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig22.Station-polaritygraphgenerationafteredgeembedding. https://doi.org/10.1371/journal.pone.0260761.g022Definition.1GivenagraphG=(V,E),whereasetoftweets’stationsV={v1,…,v16}andapredictablesetofedgesE={e1,…,eN}andNistotalnumberoftweets.Apositivesentimentpolarityprediction(p)foreachstationisalinkprediction/inferenceproblemwhereaconnectionei=vi∝pexistsiff: Lemma.Performingedgeembeddingsonthesourcedatapreventstheworst-caseiteration(i.e.,negativeorpositivesentiments)andmapsthestationpolarityfromDNNpredictiontoalinkpredictionproblem. Example.ThefollowingFigs23and24representthesentimentpolarityofdifferentstations’tweetsbeforeandafterapplyingedgeembeddingsrespectively. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig23.Twodimensions(station-polarity)graphembedding. https://doi.org/10.1371/journal.pone.0260761.g023 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig24.Attentionscoresforstations’polaritypredictions. https://doi.org/10.1371/journal.pone.0260761.g024Inadditiontothevisibilitygainedbyembeddingthegraphedges,nodeembeddings(Fig23)allowareducedrepresentationoftheobservedpolarsentimentswithaclearpolarsymmetrywithinthenewsstations.Inourcase,thegeneratedgraphconsistsofasetofnodeswhichareonlyidentifiedbytheirlabelswithoutanyotherfeatures.Asthisisnotsupportedbytherecentembeddingalgorithms(e.g.,GraphSage[85]),anabstractversionofnode2vecalgorithmhasbeenimplementedwhichinsteadofrandomlyiteratesoverallconnections,itaggregatestheneighbourhoodnodesofagivenstationfollowingthepredefinedconstraint(seeDefinition.1). Definition2.Ascoredconnectionbetweenastationandasentimentpolarityisaneighbourhoodaggregationofthescoresofitsneighbourssuchas: (oranyotherthresholdcondition)needstobeverifiedduringfeed-forwardandback-propagationstagesoftheneuralnetworkalloverthe(n)dependencies. AsshownbyFig24,scoringthepositivepolaritiesallowsatransparentconnectivityaswellasinferringnewconnections. DNNconstruction.AflexiblemannertoimplementtheabovestepsistoproceedaDNNcodingfromscratch.WithrespecttothestructuredepictedbyFig12,ithasbeenchosentousethe“Tanh”activationfunctiononthetwohiddenlayerswhichapproximatethesentimentpolarity[–1,1],theoutputlayerhasbeenactivatedbythe“Sigmoid”functionwhichscalesthepolarvectorresultingfromhiddenlayersintopositiveornegativesentiments,Where: (7) (8) Table4detailstheparametersoftheDNNstructuredepictedbyFig12,thebatchsizeofeachhiddenlayer,theactivationfunctions,theoptimizer,andtheestimatedlearningrateofeachlayer. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable4.InnerstructureparametersoftheproposedDNNcomparedtobasictechniques. https://doi.org/10.1371/journal.pone.0260761.t004AspresentedbyTable4,themodel’slearningincreasesfromtheehiddenlayers(0.027to≈0.9)bytheoutputlayer,whichconfirmsthehypothesisofthechainrule(Fig12)(i.e.,mostoflearninghappensatthedecisionalandparticularlytheoutputlevel.).TheReLuactivationfunctionhasbeenactivatingtheinputlayerasitprovidesbetterapproximationfortheembeddedfeaturesvector,wherenoclassificationhasmadeyetexceptforthefrequencyanalysis(#1inTable4),Tanhfunctionhasbestapproximationforsentimentpolarity(moredetailedonsection6,“DNNconstruction”).Sigmoidhasbeenactivatingtheoutputlayertoinferpositiveandnegativeinstances. AsmentionedbyFig25andbydisplayingthemodeltraininghistory(Fig26),ithasbeenshownarapidconvergencetoastableaccuracyof≈83%whichprovidesanansweronhowtostopthemodel’svanishingwhileitkeepspropagatingevenifitreachesanoptimalperformance. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig25.Impactofattentionscoresandembeddingsonthemodelconvergence. https://doi.org/10.1371/journal.pone.0260761.g025 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig26.Modellossandaccuracyhistory. https://doi.org/10.1371/journal.pone.0260761.g026Table5matchesthemeta-parametersinvolvedwithinthisstudywiththeirmeaningregardingthestudyingdomain. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable5.Meaningofthelearningmetrics’parameterswithregardstotheSAstudy. https://doi.org/10.1371/journal.pone.0260761.t005 Accuracyistheproportionoftrueresultsamongalltheobservedpopulation:Acc= F-measureisthemeanbetweenprecisionandrecall:F-measure= Precisionistheproportionoftrueinstancespositivelypredictedamongthetruepositiveandfalsepositiveidentifiedones.Precision= Recallreportsthepositivepolarsamplescorrectlypredictedtothetotalpositivesamples.Itreflectsthemodel’sabilitytoinferpositivesamples.Recall= ThefollowingTable6reportsthesentimentclassificationmetricsusedinthisworkandtheobtainedvalues.Wehighlightwithinthesametabletheimpactoftheproposedtechniquesonebyoneonthemodel’sperformance. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable6.Proposedmodelperformance(shownwithbold)comparedtodifferenttechniquesonhealthnewstweetsdataset. https://doi.org/10.1371/journal.pone.0260761.t006Duetothefeatures’opacity,anaiveMulti-layerDNNshowsalowaccuracy(67%)andapoorinferenceoftrueinstancespositivelypredicted(e.g.,51%precision).However,applyingthesametechniqueafterexcludingthenonrelevantfeaturesaftergraphembeddings(ISEinTable6)hasimprovedthemodel’saccuracyaswellastheprecision,buttherecall’srateremainsstable.Thisisexplainedbytheconditionalstep(see2ndpartofalgorithmabove,line11)wherethelatteronlyconsideredthepositivesentimentswhiletherecallimpliesthepositiveinstancesamongallpopulationincludingthenegativeones.Bycouplingthepreviousstepwiththescoringtechnique(adetailedexplanationisgivenin“ImprovingDNNperformanceviaadeterministicbackwardwalk”),themodelhasseenasignificantimprovementamongallmetrics,thatisjustifiedbythedeterminismgainedfromselectingrelevantfeaturesduringbackpropagation,becausethisselectioncoverstheactivationfunctions’derivatives,bothpositiveandnegativeinstanceshavebeencovered,thingthatexplainstherecallimprovement(from53%to89.5%)aswellastheothermetrics.,whichanswersthesecondpartof“Researchquestions”(RQ5). Complexityanalysis Timecomplexity.Thefollowingformula: calculatestheoverallasymptoticcomplexity(TC)ofaDNN.Byconsideringagiventhreshold(h),afeed-forwardpropagationislimitedtotheinputspaceembeddingstimesthecostoftheactivationfunctions.Inourcasetherearetwohiddenlayersactivatedwith(tanh)and(sigmoid)functionsrespectively.Suppose: TC(tanh)=O(t)andTC(sigmoid)=O(s),because(tanh)hasbiggerapproximation:O(t)>O(s) graphembeddingscomplexityisO(|V|),Visthetotalgraphnodes,therefore: Forback-propagation,thetimecomplexityisreducedtothescoringmethodwhichhas(h)asalimit,therefore:O(score)=O(Vh+Eh),fromthat: TC=O(Vh+Eh)+O(|V|)⋅O(t)whichmaybereducedtoO(|V|)⋅O(t)intheworstcase.Thelatterreflectsthenodeembeddingsstrategyadoptedbytheproposedmethod. Spacecomplexity.Insteadofstoringthematrices[94]offeaturevectorsandparameterweightsinmemoryduringtheexecutionoftheDNNmodel,theembeddedgraphentitiesaremainlysupposedtoallocatethememorywiththeactivationfunctiontraces.Atatimeinstanceepoch(i),(i=1…90)theproposedmodelhistory(e.g.,Fig26)allowstorecordthefollowingmetricssummarizedinTable7. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable7.CPUoccupancyandlearningmetricsfortheproposedmodel. https://doi.org/10.1371/journal.pone.0260761.t007ThecachehierarchyoftheCPUenablestorecordseveraltrainingbatchesoftheproposedDNN(seeTable7).Theexecutionflowshowsareducedfootprint(i.e.,3.0CPUoccupancy)resultedfromthegraphembeddingsfollowedbythebackwardscoring(seethebelowsection).Thereducedinstructionvectormayrepresentanalternativetotheindeterministicsparsitysolution[95]foranefficientDNNtraining. AsitisshownfromFig27,theCPUexperiencesabatchoftrainingandmostofitstimeonthefirstmodel’slayers(hiddenlayersfromFig27),withanaverageCPUtimeof67.6%infirsthiddenlayerto49.09%insecondone,itendswithlessCPUoccupationwithanaverageof26.7%onthedecision(output)layer.ThatjustifiesourhypothesisabouttherepetitiveworkintheinputunitofaDNN.However,themodel’saccuracyisshowntoperformreasonablywellsinceearlierneurones,that’sduetotheselectionstrategywhichpreventsfeatures’sparsityandoverfitting. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig27.AverageCPUtimeandmodel’sefficiencythrougheachlayer. https://doi.org/10.1371/journal.pone.0260761.g027 EvaluationBythissection,theimpactoftheproposedlearningmethodwillbeemphasizedthroughdifferentstages:training,learning,complexityandvalidation. Duetotheheterogeneityofthe16news’stationsandfeatures’sparsityimposedtothegeneratedgraphcomponents(i.e.,nodesareonlyidentifiedbytheirlabels),thepreliminarytests(Fig28)showalowmodelperformanceevenifitdoesnotoverfitafterembeddingtheinputspace,thelowaccuracyremainsanissueifnotimproved,becauseDNNsareknowntoperformwellwithhugedatacorpus. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig28.StabilityofproposedDNNafter90epochsand10batches. https://doi.org/10.1371/journal.pone.0260761.g028Althoughthelosshasbeensignificantlyminimized(Fig29(B)),theinstabilityremarkedwithintheaccuracy(Fig29(A))variationsremainsabottlenecktowardsthemodeladaptability. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig29.(a)Instabilityofaccuracy.(b)Lossminimization. https://doi.org/10.1371/journal.pone.0260761.g029 ImprovingDNNperformanceviaadeterministicbackwardwalk AsshownbyFigs25and30,scoringthelearningpathwhichisrecognizedwhiletrainingtheDNNmodelbecameamandatorystepinourcasestudy,inordertoimprovethewholeaccuracy.Thiswillrepresentatypicalexampleofagoodtrade-offtransparency(graphtransparency)andefficiency(DNNperformance). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig30.Neuralpath’sembedding+scoring. https://doi.org/10.1371/journal.pone.0260761.g030 Transparencyandlearningperformance.Therestrictionimposedtotheinputnodesallowedaleveloftransparencyregardingthepredictivestudy,thishasbeenreplicatedonthefeed-forwardpath,whereasdescribedbyFigs31–33,ifweconsiderpositivesentiments(polarity)as“blue”instancesandthenegativeonesas“red”ones,thedecisionboundaryshowedabetterseparationofbothpolarities.However,bestadjustmentisshownbyFig33afterscoringtheback-propagationpath(stampingpositivepolarityasaconstraint). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig31.Naïvetrainingandlearning. https://doi.org/10.1371/journal.pone.0260761.g031 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig32.Decisionboundaryafteredgeembedding. https://doi.org/10.1371/journal.pone.0260761.g032 Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig33.Impactofedgeembeddingandpathscoringonthedecisionthreshold. https://doi.org/10.1371/journal.pone.0260761.g033Consequently,resultsonadjustingthelearningcurvewithbothembeddingsandscoringmethodssequentiallywithrespecttotrainingscores(batchgradientdescent)areillustratedbyFig34. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig34.Learningimprovementswithembeddingthenscoringtechniques. https://doi.org/10.1371/journal.pone.0260761.g034TheReceiver-Operating-Characteristic(ROC)andArea-Under-the-curve(AUC)aretworelevantmetricsformodels’confidenceespeciallyinhealthcaredomain[96],thosetwometricsallowtovisualizethetrade-offbetweenthemodel’ssensitivityandspecificity,where: Sensitivity=true-positiverate(rateofcorrectlyidentifiedsentiments) Specificity=1–false-positiverate(rateofincorrectlyidentifiedsentiments) asillustratedbyFig35,theproposedlearningmodelshowedahigherAUCof94%with90%maximizationofcorrectlyidentifiedsentiments. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig35.ROCcurvesfortheproposedDNNmodel. https://doi.org/10.1371/journal.pone.0260761.g035 Comparingtoothermethods.Asapartoftheevaluation,theproposedmodeliscomparedtoseveralcomputationalframeworksrelatedtohealthcaredomainwhichaimedtoanalysetweetsandextractsentimentpolarityfollowingspecifictopics.SAwasthemosttargetedtopic[97]amongtheotherrelateddomains.However,thisprocessisstillnotdisclosed,andthefeatureextractionmechanismforsentimentclusteringisstillnotwelldefined.AsdepictedbyTable8,commonworkswhichhaveaddressedtwitterhealthnewsdatasetusedmachinelearningtechniquesforsentiments’classification.However,asarguedinthenextsection,adeepinvestigationofSArequiresdifferentapproximationswhichgobeyondlinearMLmodels. Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageTable8.Comparisonoftheproposedmethod(shownwithbold)performancewithotherapproachesontwitterhealthdataset. https://doi.org/10.1371/journal.pone.0260761.t008Ourproposedmethodshowsgreatoutcomescomparingwithothertechniques(Table7),thiscouldbeemphasizedwiththefollowingaspects: Semanticenrichment:ourproposedDNNcoversbothsentimentswithinseparatetweetsaswellasthewholetextcorpusforanoverallpolarity[–1,1]andsubjectivity[0,1],thisincludesmostfrequentterms. Complexity:acomplexityanalysishasbeenexplicitlyconducted,theasymptoticresultsfollowtheabstractionstrategy(Fig12)byrestrictingthewholemodelcomplexitytotheembeddednodestimesthecomplexityofthedecisionalfunction(Tanh).Thatperformanceismuchbetterthanconsideringallinputspaceforinstance[99]. Efficiency/determinism:AlthoughSVMhasprovenitsrobustnessandperformanceinmanySAtasks(seeTable2),itscombinationwithLSTMrepresentsabottlenecktowardsaboostedperformance.Thiscouldbejustifiedbythepre-traininganddependencycostofLSTMattheinputdata[100].However,ourproposedbackpropagationselectivestrategyincreasesthemodel’sdeterminism(i.e.,rapidsurgeofthelearningrate(Fig34)). Transparency:Ourmodelischaracterisedbyatransparentpredictiongenerationprocess,thisincludestheearlierconceptualstages(i.e.,Figs12and13)followedbyavisualdatadistributionandtheimpactoftheproposedtechniquesonbestadjustingthedecisionboundaryforsentimentclassification(Figs31,32and33).Asoppositetotheclassicalclassifiers[102],theproposedDNNstructureallowsdifferentapproximationsoftheproblem(i.e.,polarity,subjectivity,frequency,etc),thatenablesaglobalobservationoftheSAoverallthenews’stations.Thecomplianceofthebackwardselectionmethodwithbackpropagationalgorithm(see:“Features’selectionviaattentionscoring”,“ImprovingDNNperformanceviaadeterministicbackwardwalk”)doesnotrequireanyadditionaltrainingexamplesorhiddenlayersasthecasein[103],whichallowedthemodelcomplexitytoberestrictedtotheembeddedspace. Discussions ModelsonexplainableAI AlthoughDARPA’suserinterface[74]hasbeenbuiltaroundusers’expertiseandtheircognitionability,itdisguisesthetraceableaspectofthepredictionmaking,whichmayincludetheactiveneuronsandthepredictionpath. Insteadofexplaininglearningmodelsaftertheirrealization,currenttrendsinmachinelearning[104]suggestthatitismoreprominenttoincludeexplicabilityfromthefirstconceptualstepsofthemodel.However,asillustratedbyFig36,thenon-lineardistributionwhichresultsfromdistinctivefeaturescales(e.g.,Frequency[0…n],subjectivity[0…1],etc.)requiresanalternativemethodthantraditionalnonlinearMLapproximation,wherethelatterisappliedtothewholeobservations.ADNNcouldapproximateeachfeatureobservationfollowingspecificlayers,thatwhatexplainsahighersensitivityandrecallperformance(Table8). LSTMcanonlyrelateagivenaspecttothepreviousone.ButwithintheSAcontext,furtherdependenciesmayoccurandneedtobecaptured.Forinstance,in[100](seeTable8)anindexhadtobedoneinordertoboostthemodelperformance. Agoodunderstandingoftheinputdatasetcouldbeachievedbyanefficientpre-processing.However,withDNNs,thisdoesnotguaranteeagoodperformance,asthelatter(see21)isusuallyconditionedbyarandomweightassignmenttoactivatecertainfunctions.Bytheproposedmodel,weaimtomakethisprocessmoredeterministic. Dataisusuallypre-processedbeforetrainedandvalidatedbyaDLmodel,thathelpsremovingimpuritieslikestopwords,insignificance,etc.,buteventuallypromotethelossofdatainformationcentrality.Whereas,byinvestigatingagraphtheory(i.e.,embeddings)accompaniedwithaDNNdataclosenesscentralityispreserved(Fig23). Download: PPTPowerPointslidePNGlargerimageTIFForiginalimageFig36.Binarysentimentpolaritydistributionoftweets. https://doi.org/10.1371/journal.pone.0260761.g036 Limits Althoughtheproposedmodelshowedgreatconvergencewhichpreventsvanishingproblemandsavestrainingtime,itsperformancewasrelativelyweakwhendeployedonx86architecturewith5GBavailableRAM(Fig28). TheembeddingmethodpreventstheDNNtobroadthelearningscalebecausethelayersareactivatedbyproceedingtheembeddedvectoralthoughthemodelbackpropagatesthroughalltheinstances(seeAlgorithmabove)eventhoughthelossmeasureisconsiderablyless(Fig29(B)),itmainlyoptimizesthescoredweights(e.g.,positiveweights). Disclosingfeaturessemanticsin[99]hasprovenitsresiliencyinhandlingunstructureddata.Inourmodel,theembeddedfeaturevectoraswellasthescoredsamplescouldbeenrichedbyanaccompaniedcontextvectorforunderstandabilitypurposes. ConclusionandfutureworkInthisresearchwork,weaimtoproposeatransparentDNNmodelforasentimentclassifier.Ithasbeendecidedtoproceedthedevelopmentwithoutusingbuilt-inDLlibrariesexceptforevaluationmetricsinvocation,andthatwasinordertoexactlydesigneachunit:input,decisionandoutputwiththedefinedmethod(see“Methods”).Thelatterconsistsofanewperformanceimprovementstrategywhichcombinesasparsegraphembedding(i.e.,node,edgeswithnofeatures)andscoringpathsfortheinputanddecisionalunitsrespectively.ThemodelistrainedandtestedonTwitterhealthnewsdataset,whereasentimentpredictiveanalysishasbeenappliedtoeachnewssourcesbasedonthemostfrequentedtweets.Webroadthefeaturespacebynormalizingbothtokenaspectsandtweetsforeachofthe16newssothataglobalsentimentpolarityisinferred.Resultsshowstate-of-the-artperformancewhilecomparingtoothermodels(see“Predictiveanalysis”and“Comparingtoothermethods”).Moreover,thetransparencyandtheefficiencyofthemodelinstabilizingthelearningcurvewithbetterbinaryclassificationoftweets(seeabove). Thisworkcanbenefitfromseveralimprovementsinthefuture.Forinstance: Exploringthetransferablelearningaspectofgraphembeddingstoincludeotherupdatedtopicsontwitter(e.g.,Covid-19)wheremoretransparencyisrequired.Thismaybeachievedbymovingfromthetransductivetotheinductivelearning.Furthermore,thatmayprovideananswertothedynamicaspectofgraphsastheinputdatamayevolveoverthetime. Provingthemodelresiliencyagainstnewunstructuredandsemi-structureddata(SemEval-2014task7[105]). Intermsofperformance,ithasbeenproventhattheembeddingtechniquehadabigimpactonthemodelaccuracy(see“Evaluation”).Thus,byconsideringacontextfeatures’vectorwhiletrainingthemodel,thiscouldbroadthelearningstageandimprovethemodelperformance. References1. ChenL-C,LeeC-M,ChenM-Y.(2019).“Explorationofsocialmediaforsentimentanalysisusingdeeplearning”.SoftComput24,8187–8197(2020).https://doi-org.libaccess.hud.ac.uk/10.1007/s00500-019-04402-8.Accessedon14/01/202010:17. ViewArticle GoogleScholar 2. MasudM,MuhammadG,AlhumyaniH,AlshamraniSS,CheikhrouhouO,IbrahimS,etal.“Deeplearning-basedintelligentfacerecognitioninIoT-cloudenvironment”.ComputerCommunication.(2020).Volume152,15,pp.215–222,Accessedon17/05/201912:04. ViewArticle GoogleScholar 3. VyasV,UmaV.“Approachestosentimentanalysisonproductreviews”.Sentimentanalysisandknowledgediscoveryincontemporarybusiness.IGIGlobal,Pennsylvania,pp15–30. ViewArticle GoogleScholar 4. RodriguesChagasBN,NogueiraVianaJA,ReinholdO,LobatoF,JacobAFL,AltR.“CurrentapplicationsofmachinelearningtechniquesinCRM:aliteraturereviewandpracticalimplications”.IEEE/WIC/ACMIntConfWebIntell(WI)2018:452–458.https://doi.org/10.1109/WI.2018.00-53.Accessedon05/07/202122:45. ViewArticle GoogleScholar 5. RambocasM,PachecoBG.“Onlinesentimentanalysisinmarketingresearch:areview”.2020.JResInteractMark12(2):146–63. ViewArticle GoogleScholar 6. RiosN,deMendoncaNetoMG,SpinolaRO.“Atertiarystudyontechnicaldebt:types,managementstrategies,researchtrends,andbaseinformationforpractitioners”.InfSoftwTechno102:117–145. ViewArticle GoogleScholar 7. AmaraS,SubramanianRR.“Collaboratingpersonalizedrecommendersystemandcontent-basedrecommendersystemusingTextCorpus”.20206thInternationalConferenceonAdvancedComputingandCommunicationSystem(ICACCS),Coimbatore,India,2020,pp105–109.8. RozanskaA,PodporaM.“MultimodalsentimentanalysisappliedtointeractionbetweenpatientsandhumanoidrobotPepper”.IFAC-PapersOnline,2019.Accessedon22/07/202121:15. ViewArticle GoogleScholar 9. PRXJJU.“ArtificialIntelligenceinSpaceExploration”.AnalyticsVidhya.2021.Analyticsvidhya.com/blog/2021/01/artificial-intelligence-in-space-exploration/.pmid:34124465 ViewArticle PubMed/NCBI GoogleScholar 10. VoraS,MehtaRG.“InvestigatingPeople’sSentimentfromTwitterDataforSmartCities:ASurvey”.InternationalJournalofComputationalIntelligence&IoT,vol2,No2.2019. ViewArticle GoogleScholar 11. AsifM,IshtiaqA,AhmadH,AljuaidH,ShahJ.“Sentimentanalysisofextremisminsocialmediafromtextualinformation”.TelematicsInformatics48.2020.1013445. ViewArticle GoogleScholar 12. HassanSaif,MiriamFernandez&HarithAlani.“EvaluationDatasetforTwitterSentimentAnalysis”.2013.Asurveyandanewdataset,theSTS-Gold.CEURWorkshopProceedings.1096. ViewArticle GoogleScholar 13. CunliffeE,CuriniL.“ISISandheritagedestruction:Asentimentanalysis”.Antiquity,92(364),1094–1111.2018. ViewArticle GoogleScholar 14. MatalonY,MagdaciO,AlmozlinoA,etal.“Usingsentimentanalysistopredictopinioninversionintweetsofpoliticalcommunication”.2021.SciRep11,7250.pmid:33790339 ViewArticle PubMed/NCBI GoogleScholar 15. ElbattahM,ArnaudE,GignonM,DequenG.“TheRoleofTextAnalyticsinHealthcare:AReviewofRecentDevelopmentsandApplications”.InProceedingsofthe14thInternationalJointConferenceonBiomedicalEngineeringSystemsTechnologies(BIOSTEC2021). ViewArticle GoogleScholar 16. ClarkEM,JamesT,JonesCA,AlapatiA,UkanduP,DanforthCM,etal.“ASentimentAnalysisofBreastCancerTreatmentExperiencesandHealthcarePerceptionsAcrossTwitter”.2018.arXiv:1805.09959v1[cs.CL].Accessedon29/06/202113:25. ViewArticle GoogleScholar 17. GuY,CelliF,SteinbergerJ,AndersonAJ,PoesioM,StrapparavaC,etal.“UsingBrainDataforSentimentAnalysis”.JLCL2014Band29(1)–79–94. ViewArticle GoogleScholar 18. AhmadM,AftabS,BashirMS,HameedN.“SentimentAnalysisusingSVM:ASystematicLiteratureReview”.(IJACSA)InternationalJournalofAdvancedComputerScienceandApplications,vol9,No2.2018. ViewArticle GoogleScholar 19. KowsariK,JafariMeimandiK,HeidarysafaM,MenduS,BarnesL,BrownD.“Textclassificationalgorithms:Asurvey”.Information(2019),10,150. ViewArticle GoogleScholar 20. MikeT.“Genderbiasinmachinelearningforsentimentanalysis”.OnlineInformationReview;Bradford,(2018).Vol42,N°3.pp-343–354. ViewArticle GoogleScholar 21. AshokkumarP,SivaShankarG,GautamSrivastava,PraveenKumarReddyMaddikunta,andThippaReddyGadekallu.2021.“ATwo-stageTextFeatureSelectionAlgorithmforImprovingTextClassification”.ACMTrans.AsianLow-Resour.Lang.Inf.Process.20,3,Article49(April2021),19pages.https://doi.org/10.1145/3425781. ViewArticle GoogleScholar 22. ShankarGS,AshokkumarP,VinayakumarR,GhoshU,MansoorW,AlnumayWS."AnEmbedded-BasedWeightedFeatureSelectionAlgorithmforClassifyingWebDocument",WirelessCommunicationsandMobileComputing,vol.2020,ArticleID8879054,10pages,2020.https://doi.org/10.1155/2020/8879054.Accessedon25/06/202123:25.pmid:33088230 ViewArticle PubMed/NCBI GoogleScholar 23. HaqueTU,SaberNN,ShahFM.“SentimentanalysisonlargescaleAmazononproductreviews”.In2018IEEEInternationalConferenceonInnovativeResearchandDevelopment(ICIRD),(2018).pp1–6. ViewArticle GoogleScholar 24. SiemonsmaPC,BlomJW,HofstetterH,vanHespenATH,GusseklooJ,DrewesYM,etal.(2018).“Theeffectivenessoffunctionaltaskexerciseandphysicaltherapyaspreventionoffunctionaldeclineincommunitydwellingolderpeoplewithcomplexhealthproblems”.BMCGeriatr18,164.pmid:30016948 ViewArticle PubMed/NCBI GoogleScholar 25. AbualigahL,AlfarH,ShehabM,AbuHusseinAM.“SentimentAnalysisinHealthcare:ABriefReview.Inbook:RecentAdvancesinNLP:TheCaseofArabicLanguage”.(2020). ViewArticle GoogleScholar 26. YangK,ZhuJ,GuoX."POIneural-recmodelviagraphembeddingrepresentation".InTsinghuaScienceandTechnology,(2021ª).Vol26,no2,pp208–218, ViewArticle GoogleScholar 27. YueX,WangZ,HuangJ,ParthasarathyS,MoosavinasabS,HuangY,etal.“Graphembeddingonbiomedicalnetworks:methods,applicationsandevaluations”.Bioinformatics,Volume36,Issue4,15February2020,pp1241–1251,pmid:31584634 ViewArticle PubMed/NCBI GoogleScholar 28. YangJ,ZouX,ZhangW,HanH.“MicroblogsentimentanalysisviaembeddingsocialcontextsintoanattentiveLSTM”.EngineeringApplicationsofArtificialIntelligence.(2021).Vol97,104048. ViewArticle GoogleScholar 29. BijarK,ZareH,VeisiH,KebriaeiE.“LeveragingDeepGraph-BasedTextRepresentationforSentimentPolarityApplications”.ExpertSystemswithApplications.(2019).Volume144, ViewArticle GoogleScholar 30. IvakhnenkoAG,LapaVG.(1965).CyberneticPredictingDevices.CCMInformationCorporation.NewYork:CCMInformationCorp.pmid:1434529931. MinskyM.(1963).“Stepstowardartificialintelligence””.Computersandthought,McGraw-Hill,NewYork,pp406–450.https://doi.org/10.1037/h0040616pmid:1408679132. AlazabM,KhanS,RamaKrishnanSS,PhamQ-V,KumarReddyMP,ReddyGadekalluTR.“AMultidirectionalLSTMModelforPredictingtheStabilityofaSmartGrid”.Vol8,2020.Accessedon26/06/202107:45. ViewArticle GoogleScholar 33. LillicrapTP,SantoroA.“Backpropagationthroughtimeandthebrain.CurrentOpinioninNeurobiology”.(2019).Vol55,pp82–89.pmid:30851654 ViewArticle PubMed/NCBI GoogleScholar 34. GuoY,ChenJ,DuQ,V-DHengelA,ShiQ,TanM.“Multi-waybackpropagationfortrainingcompactdeepneuralnetworks”.NeuralNetworks.Volume126,June2020,pp250–261.pmid:32272429 ViewArticle PubMed/NCBI GoogleScholar 35. SmithRC.“It’sTimetoViewSevereMedicallyUnexplainedSymptomsasRed-FlagSymptomsofDepressionandAnxiety”.JAMANetwOpen.(2020).3(7):e2011520.pmid:32701154 ViewArticle PubMed/NCBI GoogleScholar 36. HuangW,RaoG,FengZ,CongQ.“LSTMwithsentencerepresentationsfordocument-levelsentimentclassification”.Neurocomputing,(2018).308:49. ViewArticle GoogleScholar 37. ViolosJ,TserpesK,PsomakelisE,PsychasK,VarvarigouTA.(2016).“Sentimentanalysisusingword-graphs”.InWIMS,p_22. ViewArticle GoogleScholar 38. ZhaoP,HouL,WuO.“Modelingsentimentdependencieswithgraphconvolutionalnetworksforaspect-levelsentimentclassification”.Knowledge-BasedSystems.Volume193,105443.https://doi.org/10.1016/j.knosys.2019.105443. ViewArticle GoogleScholar 39. VeličkovićP,CucurullG,CasanovaA,RomeroA,LioP,BengioY.“GraphAttentionNetworks”.MachineLearning(stat.ML).2017.ArXiv:1710.10903[stat.ML]. ViewArticle GoogleScholar 40. ShmidhuberJ.“DeepLearninginNeuralNetworks:AnOverview”.TechnicalReportIDSIA-03-14/;2014,arXiv:1404.7828v3[cs.NE]. ViewArticle GoogleScholar 41. SinghA,SenguptaS,LakshminarayananV.“Explainabledeeplearningmodelsinmedicalimageanalysis”.2020;arXiv:2005.13799v1[cs.CV].pmid:34460598 ViewArticle PubMed/NCBI GoogleScholar 42. RichardsBA,LillicrapTP,BeaudoinP,BengioY,BogaczR,ChristensenA,etal.“Adeeplearningframeworkforneuroscience”.NatNeurosci.2019Nov;22(11):1761–1770.Epub2019Oct28.pmid:31659335;PMCID:PMC7115933. ViewArticle PubMed/NCBI GoogleScholar 43. SunJ,BinderA."GeneralizedPatternAttributionforNeuralNetworkswithSigmoidActivations".InternationalJointConferenceonNeuralNetworks(IJCNN),Budapest,Hungary,2019;pp1–9,https://doi.org/10.1109/IJCNN.2019.885176144. WuY,ZhangS,ZhangY,BengioY,SalakhutdinovRR.“Onmultiplicativeintegrationwithrecurrentneuralnetworks”.InAdvancesinNeuralInformationProcessingSystems,2016;pp2856–2864. ViewArticle GoogleScholar 45. KumarS,SharmaA,TsunodaT.“Brainwaveclassificationusinglongshort-termmemorynetworkbasedOPTICALpredictor”.SciRep9,9153.2019;pmid:31235800 ViewArticle PubMed/NCBI GoogleScholar 46. LiB,ChengZ,XuZ,YeW,LukasiewiczT,ZhangS.“Longtextanalysisusingslicedrecurrentneuralnetworkswithbreakingpointinformationenrichment”.In:Proceedingsofthe2019IEEEinternationalconferenceonacoustics,speechandsignalprocessing,ICASSP2019.Vol124,pp51–60. ViewArticle GoogleScholar 47. LiuYH,SmithS,MihalasS,Shea_BrownE,SümbülY.“Asolutiontotemporalcreditassignmentusingcell-type-specificmodulatorysignals”.BioRxiv.2020;https://doi.org/10.1101/2020.11.22.393504. ViewArticle GoogleScholar 48. WeberT,HeessN,BuesingL,SilverD.“CREDITASSIGNMENTTECHNIQUESINSTOCHASTICCOMPUTATIONGRAPHS”.2019;arXiv:1901.01761v1[cs.LG]. ViewArticle GoogleScholar 49. GoyalA,KeNR,LambA,HjelmRD,PalC,PineauJ,etal.“ACTUAL:ACTOR-CRITICUNDERADVERSARIALLEARNING”.2017,arXiv:1711.04755v1[stat.ML]. ViewArticle GoogleScholar 50. GraziotinMD,KuutilaM.“Theevolutionofsentimentanalysis—Areviewofresearchtopics,venues,andtopcitedpapers”.ComputerScienceReview,(2018),Vol27,pp16–32,ISSN1574-0137,https://doi.org/10.1016/j.cosrev.2017.10.002. ViewArticle GoogleScholar 51. TorresAD,YanH,AboutalebiAH,DasA,DuanL,RadP.“Chapter3—PatientFacialEmotionRecognitionandSentimentAnalysisUsingSecureCloudWithHardwareAcceleration”.Intelligentdata-Centricsystems.2018.pp61–89, ViewArticle GoogleScholar 52. ZunicA,CorcoranP,&SpasicI.“SentimentAnalysisinHealthandWell-Being:SystematicReview”.JMIRmedicalinformatics,2020,8(1),e16023.pmid:32012057 ViewArticle PubMed/NCBI GoogleScholar 53. AravantinoV,DiehlF.“TraceabilityofDeepNeuralNetworks.MachineLearning(cs.LG)”.(2018).arXiv:1812.06744[cs.LG]. ViewArticle GoogleScholar 54. YinY,SongY,ZhangM.“Document-levelmulti-aspectsentimentclassificationasmachinecomprehension”.In:Proceedingsofthe2017conferenceonempiricalmethodsinnaturallanguageprocessing,pp2044–2054,http://www.cse.ust.hk/~yqsong/papers/2017-EMNLP-AspectClassification.pdf. ViewArticle GoogleScholar 55. HuangY,JinW,YuZ,LiB.“SupervisedfeatureselectionthroughDeepNeuralNetworkswithpairwiseconnectedstructure”.Knowledge-BasedSystems,2020,Vol204,106202,https://doi.org/10.1016/j.knosys.2020.106202. ViewArticle GoogleScholar 56. KrausM,FeuerriegelS.“Sentimentanalysisbasedonrhetoricalstructuretheory:learningdeepneuralnetworksfromdiscoursetrees”.ExpertSystAppl,(2019),118:65–79. ViewArticle GoogleScholar 57. MaasAL,DalyRE,PhamPT,HuangD,NgAY,PottsC.“Learningwordvectorsforsentimentanalysis”.In:Proceedingsof49thannualmeetingoftheAssociationforComputationalLinguistics:HumanLanguageandTechnology,2011.pp142–150. ViewArticle GoogleScholar 58. ArulmuruganR,SabarmathiKR,AnandakumarH.“Classificationofsentencelevelsentimentanalysisusingcloudmachinelearningtechniques”.ClusterComput22,1199–1209.2019,https://doi-org.libaccess.hud.ac.uk/10.1007/s10586-017-1200-1. ViewArticle GoogleScholar 59. SongM,ParkH,ShinK-s.“Attention-basedlongshort-termmemorynetworkusingsentimentlexiconembeddingforaspect-levelsentimentanalysisinKorean”.InformationProcessingandEManagement,2019,Vol56,Issue3,pp637–653. ViewArticle GoogleScholar 60. ReZ,ZengG,ChenL,ZhangQ,ZhangC,PanD."ALexicon-EnhancedAttentionNetworkforAspect-LevelSentimentAnalysis,"inIEEEAccess,2020,vol.8,pp93464–93471, ViewArticle GoogleScholar 61. YouQ,CaoL,JinH,LuoJ,“Robustvisual-textualsentimentanalysis:Whenattentionmeetstree-structuredrecursiveneuralnetworks,”inProc.ACMMultimedia,2016,pp.1008–1017. ViewArticle GoogleScholar 62. ChenF,JiR,SuJ,CaoD,GaoY.“Predictingmicroblogsentimentsviaweaklysupervisedmultimodaldeeplearning”.IEEETransMultimed.2018,20(4):997–1007. ViewArticle GoogleScholar 63. DengJ,etal.(2009).“ImageNet:Alarge-scalehierarchicalimagedatabase,”inProc.IEEEConf.Comput.Vis.PatternRecogn,pp248–255. ViewArticle GoogleScholar 64. XueW,ZhouW,LiT,WangQ.“MTNA:Aneuralmulti-taskmodelforaspectcategoryclassificationandaspecttermextractiononrestaurantreview”.ProceedingsoftheEighthInternationalJointConferenceonNaturalLanguageProcessing.2017,(Volume2:ShortPapers),2,pp151–156. ViewArticle GoogleScholar 65. AgarwalA,YadavA,VishwakarmaDK.“MultimodalsentimentanalysisviaRNNvariants”.InIEEEinternationalconferenceonbigdata,cloudcomputing,datascienceandengineering(BCD),2019,pp19–23. ViewArticle GoogleScholar 66. ZadehA,ZellersR,PincusE,MorencyL."MOSI:MultimodalCorpusofSentimentIntensityandSubjectivityAnalysisinOnlineOpinionVideos",IEEEIntell,Syst,2016. ViewArticle GoogleScholar 67. PandeyaYR,LeeJ.“Deeplearning-basedlatefusionofmultimodalinformationforemotionclassificationofmusicvideo”.MultimediaToolsandApplications.2021,80(2),pp2887–2905. ViewArticle GoogleScholar 68. El-AffendiM,AlrajhiK,HussainA.“ANovelDeepLearning-BasedMultilevelParallelAttentionNeural(MPAN)ModelforMultidomainArabicSentimentAnalysis”,inIEEEAccess,vol.9,pp7508–7518,2021. ViewArticle GoogleScholar 69. WANGX,WUP,LIUG,HUANGQ,HUX,XUH.“LearningperformancepredictionviaconvolutionalGRUandexplainableneuralnetworksine-learningenvironments”.Computing,ArchivesforInformaticsandNumericalComputation,2019,101(6),pp587–604. ViewArticle GoogleScholar 70. YangF,ZhangW,TaoL,MaJ.“TransferLearningStrategiesforDeepLearning-basedPHMAlgorithms”.Appl.Sci.2020,10,2361,2020; ViewArticle GoogleScholar 71. SeoD,OhK,OhI."RegionalMulti-ScaleApproachforVisuallyPleasingExplanationsofDeepNeuralNetworks,"inIEEEAccess,vol.8,pp8572–8582,2020; ViewArticle GoogleScholar 72. KimB,ParkJ,SuhJ.“TransparencyandaccountabilityinAIdecisionsupport:Explainingandvisualizingconvolutionalneuralnetworksfortextinformation”.DecisionSupportSystems.Vol134,11330.2020;https://doi.org/10.1016/j.dss.2020.113302. ViewArticle GoogleScholar 73. YuanH,TangJ,HuX,JiS.“XGNN:TowardsModel-LevelExplanationsofGraphNeuralNetworks”,2020°,arXiv:2006.02587v1[cs.LG]. ViewArticle GoogleScholar 74. SheL,ChaiJY.“InteractiveLearningforAcquisitionofGroundedVerbSemanticstowardsHuman-RobotCommunication”.InProceedingsofthe55thAnnualMeetingoftheAssociationforComputationalLinguistics,2017,vol.1,1634–44.Stroudsburg,PA:AssociationforComputationLinguistics.https://doi.org/10.18653/v1/P17-115075. IslamM,AndersonDT,PinarAJ,HavensTC,ScottG,KellerJM."EnablingExplainableFusioninDeepLearningWithFuzzyIntegralNeuralNetworks”.InIEEETransactionsonFuzzySystems.2020;Vol28,no7,pp1291–1300, ViewArticle GoogleScholar 76. KojimaR,IshidaS,OhtaM.,etal.“kGCN:agraph-baseddeeplearningframeworkforchemicalstructures”.J-Cheminform,12,32.,2020.pmid:33430993 ViewArticle PubMed/NCBI GoogleScholar 77. YangZ,DongS.“HierarchicalAttentionGraphConvolutionalNetworkIncorporatingKnowledgeGraphforExplainableRecommendation”.Knowledge-BasedSystems2020,Volume204,106194, ViewArticle GoogleScholar 78. SelvarajuRR.,CogswellM,DasA,VedantamR,ParikhD,BatraD.“Grad-CAM:Visualexplanationsfromdeepnetworksviagradient-basedlocalization”.InProceedingsoftheIEEEInternationalConferenceonComputerVision,2017;618–626. ViewArticle GoogleScholar 79. ZhuQ,XuY,WangH,ZhangC,HanJ,YangC.“TRANSFERLEARNINGOFGRAPHNEURALNETWORKSWITHEGO-GRAPHINFORMATIONMAXIMIZATION”,2020.arXiv:2009.05204v1[cs.LG]. ViewArticle GoogleScholar 80. GreffK,SrivastavaRK,KoutníkJ,SteunebrinkBR,SchmidhuberJ."Lstm:Asearchspaceodyssey",IEEEtransactionsonneuralnetworksandlearningsystems,2017;vol28,10,pp2222–2232.pmid:27411231 ViewArticle PubMed/NCBI GoogleScholar 81. AhnH,YimC.“ConvolutionalNeuralNetworksUsingSkipConnectionswithLayerGroupsforSuper-ResolutionImageReconstructionBasedonDeepLearning”.Appl.Sci.10,1959.2020; ViewArticle GoogleScholar 82. ZhuJ,MengQ,ChenW,MaZ.“InterpretingBasisPathSetinNeuralNetworks”,2020,https://arxiv.org/pdf/1910.09402. ViewArticle GoogleScholar 83. YuanH,JiS.“StructPool:StructuredGraphPoolingviaConditionalRandomFields”.In-internationalConferenceonLearningRepresentations.(2020ª).Availablefromhttps://openreview.net/forum?id=BJxg_hVtwH. ViewArticle GoogleScholar 84. ZhangW,YueX,LinW,WuW,LiuR,HuangF,etal.“Predictingdrug-diseaseassociationsbyusingsimilarityconstrainedmatrixfactorization”.BMCBioinformatics19,233,2018.pmid:29914348 ViewArticle PubMed/NCBI GoogleScholar 85. HamiltonWL,YingR,LeskovecJ.“InductiveRepresentationLearningonLargeGraphs”.31stConferenceonNeuralInformationProcessingSystems(NIPS2017),LongBeach,CA,USA.86. ZhouZ,LiX,N.ZareR.“OptimizingChemicalReactionswithDeepReinforcementLearning”.ACSCent.Sci.2017,3,1337−1344.pmid:29296675 ViewArticle PubMed/NCBI GoogleScholar 87. HatamiN,GavetY,DebayleJ.“ClassificationofTime-SeriesImagesUsingConvolutionalNeuralNetworks”.2017.arXiv:1710.00886v2[cs.CV].01/07/202121:32.pmid:28558002 ViewArticle PubMed/NCBI GoogleScholar 88. DubeS.“HighDimensionalSpaces.DeepLearningandAdversarialExamples”.2018.arXiv:1801.00634v1[cs.CV].Accesseson14/07/202116:21. ViewArticle GoogleScholar 89. Lv-dMaaten,HintonG.“VisualizingDatausingt-SNE”.2008.JournalofMachineLearningResearch9(2008)2579–2605.Accessedon29/06/202123:56. ViewArticle GoogleScholar 90. KaramiA,GnagopadhyayA,ZhouB,KharraziH.“Fuzzyapproachtopicdiscoveryinhealthandmedicalcorpora.InternationalJournalofFuzzySystems”,2017,pp1–12. ViewArticle GoogleScholar 91. RajputNK,GroverBA,RathiVK,“WORDFREQUENCYANDSENTIMENTANALYSISOFTWITTERMESSAGESDURINGCORONAVIRUSPANDEMIC”.2020,arXiv:2004.03925v1[cs.IR]. ViewArticle GoogleScholar 92. WaheebSA,AhmedKhanN,ChenB,ShangX.“MachineLearningBasedSentimentTextClassificationforEvaluatingTreatmentQualityofDischargeSummary”.Information.2020;11(5):281. ViewArticle GoogleScholar 93. AroraR,BasuA,MianjyP,MukherjeeA.“UnderstandingDeepNeuralNetworkswithRectifiedLinearUnits”.ICLR2028.arXiv:1611.01491v6[cs.LG]. ViewArticle GoogleScholar 94. DongX,ZhouL.“Deepnetworkasmemoryspace:complexity,generalization,disentangledrepresentationandinterpretability”.2019,arXiv:1907.06572v1[cs.LG].Accessedon21/06/502117:28. ViewArticle GoogleScholar 95. HoeflerT,listarhD,Ben-NunT,DrydenN,PesteA.“SparsityinDeepLearning:Pruningandgrowthforefficientinferenceandtraininginneuralnetworks”.2021.arXiv:2102.00554v1[cs.LG].Accessedon04/06/202114:25. ViewArticle GoogleScholar 96. NamdarK,HaiderMA,KhavatiF.“AModifiedAUCforTrainingConvolutionalNeuralNetworks:TakingConfidenceintoAccount”.2020,ArXiv:2006.04836[cs.LG]. ViewArticle GoogleScholar 97. KaramiA,LundyM,WebbF,DwivediYK."TwitterandResearch:ASystematicLiteratureReviewThroughTextMining,"inIEEEAccess,2020;vol.8,pp67698–67717. ViewArticle GoogleScholar 98. ShawG,KaramiA.“ComputationalContentAnalysisofNegativeTweetsforObesity,Diet,Diabetes,andExercise”.ASIS&T2017,Washington,DC. ViewArticle GoogleScholar 99. KaramiA,GangopadhyayA,ZhouB,KharraziH.“FuzzyApproachTopicDiscoveryinHealthandMedicalCorpora”,2017;arXiv:1705.00995v2[stat.ML].Accessedon08/06/202109:47. ViewArticle GoogleScholar 100. JiangK,FengS,SongQ,CalixRA,GuptaM,BernardGN.“IdentifyingtweetsofpersonalhealthexperiencethroughwordembeddingandLSTMneuralnetwork”.BMCBioinformatics2018,19,210.Accessedon12/06/202107:14.pmid:29897323 ViewArticle PubMed/NCBI GoogleScholar 101. KolajoT,KolajoJO.“SENTIMENTANALYSISONTWITTERHEALTHNEWS”.FUDMAJournalofScience(FJS).2018,Vol.2No.2,pp14–20. ViewArticle GoogleScholar 102. CirqueiraD,AlmeidaF,CakirG,JacobA,LobatoF,BezbradicaM,etal.“ExplainableSentimentAnalysisApplicationforSocialMediaCrisisManagementinRetail”.InProceedinsofthe4thInternationalConferenceonComputer-HumanInteractionResearchandApplications(CHIRA2020),pp319–328. ViewArticle GoogleScholar 103. ChenH,JiY.“ImprovingtheExplainabilityofNeuralSentimentClassifiersviaDataAugmentation”.arXiv:1909.04225v4[cs.CL].Accessedon02/06/202112:25. ViewArticle GoogleScholar 104. RudinC.“Stopexplainingblackboxmachinelearningmodelsforhighstakesdecisionsanduseinterpretablemodelsinstead”.NatureMachineIntelligence,1(5),206;2019. ViewArticle GoogleScholar 105. PradhanS,ElhadadNChapmanW,ManandharS,SavovaG.(2014).SemEval-2014Task7:AnalysisofClinicalText”.Proceedingsofthe8thInternationalWorkshoponSemanticEvaluation,2014;pp54–62. ViewArticle GoogleScholar DownloadPDF   Citation XML Print Printarticle Reprints Share Reddit Facebook LinkedIn Mendeley Twitter Email   Advertisement SubjectAreas? FormoreinformationaboutPLOSSubjectAreas,click here. Wewantyourfeedback.DotheseSubjectAreasmakesenseforthisarticle?ClickthetargetnexttotheincorrectSubjectAreaandletusknow.Thanksforyourhelp! Graphs   IstheSubjectArea"Graphs"applicabletothisarticle? Yes No Thanksforyourfeedback. Twitter   IstheSubjectArea"Twitter"applicabletothisarticle? Yes No Thanksforyourfeedback. Neuralnetworks   IstheSubjectArea"Neuralnetworks"applicabletothisarticle? Yes No Thanksforyourfeedback. Semantics   IstheSubjectArea"Semantics"applicabletothisarticle? Yes No Thanksforyourfeedback. Neurons   IstheSubjectArea"Neurons"applicabletothisarticle? Yes No Thanksforyourfeedback. Machinelearning   IstheSubjectArea"Machinelearning"applicabletothisarticle? Yes No Thanksforyourfeedback. Algorithms   IstheSubjectArea"Algorithms"applicabletothisarticle? Yes No Thanksforyourfeedback. Learning   IstheSubjectArea"Learning"applicabletothisarticle? Yes No Thanksforyourfeedback.