Solution 1 :

When you are looking at the HTML using F12 / Dev Tools, HTML you see is very different from what you see in HtmlAgilityPack or any other web scraping tool.


Your code doesnt work and wont work because there are only two div tags in the entire document. /html/body/div will work because there are two of these, and thats it. Rest is just js scripts.

When you load the URL in chrome, chrome compiles the data, executes the scripts and then present the data that it rendered to show you what you are supposed to see.

The URL you provided only has scripts in its body that execute and generate the divs you are seeing in the Dev / Tools and at this time, HTML Agility Pack is NOT able to execute the scripts and render a compiled HTML for you to scrape through.

What you get in HTMLAgilityPack

When you look at the code in the doc.DocumentNode, you only see this

<div id="app">
    WebUntis wird geladen ...

Chrome / IE will load something else because thats after compilation / rendering. What you are looking to do is to run the scripts in HTMLAgilityPack.. which is not something that you can do at this time.

What you see in Chrome / Browser

<div id="app">
    <div style="height: 100%;">
        <div class="un-app">
            <nav class="un-app-header navbar navbar-default">
                <div class="container-fluid">

Problem :

This is my code:

var html = webBrowser1.DocumentText;

            HtmlWeb web = new HtmlWeb();

            var htmlDoc = new HtmlAgilityPack.HtmlDocument();

            var node = htmlDoc.DocumentNode.SelectSingleNode("/html/body/div/div/div/div/section/section/div/div/div/div").Attributes["class"].Value;

            Console.WriteLine("Node Name: " + node);

So far everything works fine, but if I add a “/ div” to “SelectSingleNode” then it won’t work (error message: “Exception thrown:” System.NullReferenceException “”), although there is another “div” in the HTML code there.

I think it is because in the HTML code before the next “div” there is a “:: before”, but only if i analyze it in the browser

A part of the HTML code:

 <div class="un-page__body">
    <div class="container-fluid">
    <div class="row">
       <div class="col-sm-6">


Comment posted by Minimal Reproducible example

Welcome to SO! Please review and edit your question with a

Comment posted by Jawad

Any reason why you have to go down the path of using the entire xpath? Have you considered using // (to search within doc) like

Comment posted by Game Lion

@Jawad yes //div[@class=’className’] doesnt work. I think the website does have a mechanicm to block this

Comment posted by Game Lion

I get an answer for all “div” that are before the “div” with the class “row” and for the “div” with the class “row”, but as soon as I after the “div” with the class “col-sm -6 “search and all” div “after that I get an error

Comment posted by Jawad

Can you provide the html you are scraping or url

Comment posted by Game Lion

Thank you so much for your answer. Do you know another way to login on this page?

Comment posted by Jawad

Unless that site offers an api to sign in with, I am not sure it would be possible

Comment posted by Jawad

@GameLion, does this answer your question on why you are not able to see the data? Please do mark the post as answered if it did