1.  tez  complie 및 tar.gz 생성 
     git clone https://github.com/apache/tez
     cd tez
     pom.xml 을 수정
     <hadoop.version>2.7.1</hadoop.version>
     <pig.version>0.15.0</pig.version>
     mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true  (pig 0.15.0 의 경우  tez 0.7.0 , hadoop 2.6.0 기본 변경이 필요한 경우 수정하여 재Build )

2. Hadoop 적용
$hadoop fs -mkdir /apps/tez
$hadoop fs -put tez-dist/target/tez-0.7.0.tar.gz /apps/tez    (tez lib 들은 hdfs 에 존재해야 함)
mkdir ~/programs/tez
~/programs/tez/tez-site.xml                    
  • tez-site.xml
<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>

     <name>tez.lib.uris</name>

        <value>${fs.defaultFS}/user/xxxx/apps/tez-0.7.0.tar.gz</value>

  </property>

  <property>

    <description>URL for where the Tez UI is hosted</description>

    <name>tez.tez-ui.history-url.base</name>

    <value>http://xxxxx:8080/tez-ui/</value>

  </property>

  <property>

    <name>tez.allow.disabled.timeline-domains</name>

    <value>true</value>

  </property>

  <property>

    <description>Enable Tez to use the Timeline Server for History Logging</description>

    <name>tez.history.logging.service.class</name>

    <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>

  </property>

</configuration>
  • vi .bashrc

export TEZ_HOME=~/programs/tez
export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:~/programs/config/tez

  • tez-ui

shell>tar -zxvf apache-tomcat-7.0.50.tar.gz
shell>ln -s ../packages/apache-tomcat-7.0.50/
shell>cd webapps
shell>mkdir tez-ui
shell>cd tez-ui
shell>cp ~/packages/tez-0.7.0/tez-ui-0.7.0.war .
shell>jar xvf tez-ui-0.7.0.war
shell>cd scripts
shell>vi config.js

App.setConfigs({

  /* Environment configurations */
  envDefaults: {
    version: "0.7.0",
    /*
     * By default TEZ UI looks for timeline server at http://localhost:8188, uncomment and change
     * the following value for pointing to a different domain.
     */
     timelineBaseUrl: 'http://xxxxxx:0000',

    /*
     * By default RM web interface is expected to be at http://localhost:8088, uncomment and change
     * the following value to point to a different domain.
     */
     RMWebUrl: 'http://xxxxxx:0000',
  },


vi .bashrc 에 아래내용을 추가

export TEZ_HOME=~/programs/tez
export PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:~/programs/config/tez


  • yarn-site.xml
<property>

  <description>Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by end-users will post entities and events to the Timeline server.</description>

  <name>yarn.timeline-service.enabled</name>

  <value>true</value>

</property>

<property>

  <description>The hostname of the Timeline service web application.</description>

  <name>yarn.timeline-service.hostname</name>

  <value>master.xxxxxxxx</value>

</property>

<property>

  <description>Enables cross-origin support (CORS) for web services where cross-origin web response headers are needed. For example, javascript making a web services request to the timeline server.</description>

  <name>yarn.timeline-service.http-cross-origin.enabled</name>

  <value>true</value>

</property>

<property>

  <description>Publish YARN information to Timeline Server</description>

  <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>

  <value>true</value>

</property>

      <property>

       <description>The http address of the Timeline service web application.</description>

       <name>yarn.timeline-service.webapp.address</name>

       <value>master.xxxxxx:0000</value>

     </property>

<property>

   <description>The https address of the Timeline service web application.</description>

  <name>yarn.timeline-service.webapp.https.address</name>

  <value>master.xxxxxx:0000</value>

</property>
  • tez-ui 기동
tomcat/bin/startup.sh

참고:

pig/lib/h2/ 밑에 tez 관련 jar 파일들 이 존재


            



                    

  



  



  



  




                    


  
'Hadoop > Pig' 카테고리의 다른 글

  

    

      tez 0.7.0 에서 0.8.2 로 업그레이드  (0)
      2016.08.03
    

    

      Pig & Tez Error  (0)
      2016.08.03
    

    

      Pig 재빌드  (0)
      2016.08.03
    

    

      pig  centos 하일라이팅  (0)
      2016.08.03
    

  






    


    

    

  


          
      
          
  

  

    

      
    


    

                  



            

1. Tez  모드로 pig 실행시 plan관련 error
  org.apache.tez.dag.api.TezUncheckedException: org.apache.pig.impl.plan.VisitorException: ERROR 0: java.lang.NullPointerException

대응

  • tez 0.7.0 에서 parallelism default 값이 true 로 되어 있음 해당값을 false 로 처리 
          pig -Dpig.tez.auto.parallelism=false -x tez

2. Script  Parsing Error

예약어 관련 에러 
datetime 

대응



            



                    

  



  



  



  




                    


  
'Hadoop > Pig' 카테고리의 다른 글

  

    

      tez 0.7.0 에서 0.8.2 로 업그레이드  (0)
      2016.08.03
    

    

      Tez 설치  (0)
      2016.08.03
    

    

      Pig 재빌드  (0)
      2016.08.03
    

    

      pig  centos 하일라이팅  (0)
      2016.08.03
    

  






    


    

    

  


          
      
          
  

  

    

      
    


    

                  



            
If you are using Hadoop 0.23.X or 2.X, please add -Dhadoopversion=23 in your ant command line in the previous steps

Hadoop 2.X 의 경우 -Dhadoopversion=23 를 붙여 재 빌더
shell>ant -Dhadoopversion=23


Pig 와 Hive의 차이


            



                    

  



  



  



  




                    


  
'Hadoop > Pig' 카테고리의 다른 글

  

    

      tez 0.7.0 에서 0.8.2 로 업그레이드  (0)
      2016.08.03
    

    

      Tez 설치  (0)
      2016.08.03
    

    

      Pig & Tez Error  (0)
      2016.08.03
    

    

      pig  centos 하일라이팅  (0)
      2016.08.03
    

  






    


    

    

  


          
      

          

          

          
            

              
+ Recent posts

              
    
                
                
    
              

            

          

          


          
        
 
      
 
    
 
  

  
  
  
  
  


  
티스토리툴바