DNS problem in BIG DATA

In Big data ecosystem all the nodes have hostname in a cluster, when we are trying to access ResoureManager, NodeManager, Application Master,Job History server, Spark UI, SparkHistroyServer we always change the hostname to IP address, each and every time we are doing it, especially when we trying to check job progress, metrics, job logs we are navigating from ResourceManager to ApllicationMaster/HistoryServer. Believe me its a headache to change it each and every time.

There is a solution to overcome this problem. Follow below simple steps 
  1. In Windows machine, open Notepad++ as Admin user
  2. Browse C:\Windows\System32\drivers\etc\hosts this file, edit file in Notepad++.
  3. Add ipaddress hostname to the file 
  4. Save that file, now you can able to access the website without need to change hostname to IP address. 
Refer below hosts file opened in Notepad++ with highlighted area those entry are added now to access
windows_hosts_file

Comments

  1. Valentin Kovalenko26 June 2020 at 21:19

    "we always change the hostname to IP address" - why would you do this and what does it have to do with "big data"?

    ReplyDelete
    Replies
    1. sree21 May 2021 at 12:07

      Normally companies will use private ip to communicate within a cluster of nodes for big data hadoop ecosystem. so the privateip is mapped to hostname, not with the public ip.,
      which is why we need to map hostname to public ip.

      Delete

Post a Comment

Popular posts from this blog

Avro Secondary Sorting

In this post, We are gonna see secondary sorting with Avro file format, using AvroJob Api and its gonna be two part series, MapReduce V1 Api (org.apache.hadoop.mapred) MapReduce V2 Api (org.apache.hadoop.mapreduce) In this post we are going to see example in MapReduce V1 Api. So lets start, Here's the Avro  Input Schema that we are going to use in this example. Sample Data which we are going to read as input file, for viewing purpose data is shown in JSON format. but it was read as  avro format in mapreduce program. Secondary sorting means the values sent to the Reducer should be sorted based on some criteria. you can refer more about secondary sorting here . In this example, Reducer Iterator values should be sorted based the TimeStamp of above data, but the Reducer Input Key should be grouped based on Id of above sample data. We need use MapOutput Key Schema for Secondary Sorting(Natural Key + Composite Key).Here Natural Key is Id and Composite Ke...
Read more

Builder Design Pattern

Image
         Builder design patter is used to create an object with optional instance attributes.If you want some introduction about design patterns please refer here.         So, first step is to know what wikipedia says about builder design pattern, The intent of the Builder design pattern is to separate the construction of a complex object from its representation.         Second Step is to find an real world example (behavior) of builder design pattern. Builder pattern is similar to Buffet system, where diners will serve themselves (they pick what dishes they want) , here we need map each dish as instance attributes and people as the client class need to create a object with optional attributes.                 Now the fun part, below codes shows the implementation of builder pattern         Here's th...
Read more

Git Bundle

       Keeping git repositories in sync is usually and easily done using git pull and git push . However, for those rare times when you don't have network connectivity (or limited connectivity) and still needs to transfer the contents of a repository to somewhere else, there's git bundle .       Another use case, would be like you have hired external contractors to build some application, now you need to get code from them to your organization internal private Git Server hosted and maintained by your organization  and it doesn't exposed to public. So instead of sending .zip format of code, they can send git bundle. What Git Bundle Covers?       Git bundle file is a full repository in a single file, you can have branches , commit history , tags , basically everything you expect in a repository, but it's all contained in a single file.This makes sharing the repository or moving the full repository pretty ...
Read more